System Having a Hybrid Threading Processor, a Hybrid Threading Fabric Having Configurable Computing Elements, and a Hybrid Interconnection Network

ABSTRACT

Representative apparatus, method, and system embodiments are disclosed for configurable computing. In a representative embodiment, a system includes an interconnection network, a processor, a host interface, and a configurable circuit cluster. The configurable circuit cluster may include a plurality of configurable circuits arranged in an array; an asynchronous packet network and a synchronous network coupled to each configurable circuit of the array; and a memory interface circuit and a dispatch interface circuit coupled to the asynchronous packet network and to the interconnection network. Each configurable circuit includes instruction or configuration memories for selection of a current data path configuration, a master synchronous network input, and a data path configuration for a next configurable circuit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of andpriority to U.S. patent application Ser. No. 17/372,439, filed Jul. 10,2021, inventor Tony M. Brewer, titled “System Having a Hybrid ThreadingProcessor, a Hybrid Threading Fabric Having Configurable ComputingElements, and a Hybrid Interconnection Network”, which is a continuationof and claims the benefit of and priority to U.S. patent applicationSer. No. 16/176,434, filed Oct. 31, 2018 and issued Aug. 17, 2021 asU.S. Pat. No. 11,093,251 B2, inventor Tony M. Brewer, titled “SystemHaving a Hybrid Threading Processor, a Hybrid Threading Fabric HavingConfigurable Computing Elements, and a Hybrid Interconnection Network”,which is a nonprovisional of and claims the benefit of and priority to(1) U.S. Provisional Patent Application No. 62/579,749, filed Oct. 31,2017; (2) U.S. Provisional Patent Application No. 62/651,128, filed Mar.31, 2018; (3) U.S. Provisional Patent Application No. 62/651,131, filedMar. 31, 2018; (4) U.S. Provisional Patent Application No. 62/651,132,filed Mar. 31, 2018; (5) U.S. Provisional Patent Application No.62/651,134, filed Mar. 31, 2018; (6) U.S. Provisional Patent ApplicationNo. 62/651,135, filed Mar. 31, 2018; (7) U.S. Provisional PatentApplication No. 62/651,137, filed Mar. 31, 2018; (8) U.S. ProvisionalPatent Application No. 62/651,140, filed Mar. 31, 2018; (9) U.S.Provisional Patent Application No. 62/651,142, filed Mar. 31, 2018; (10)U.S. Provisional Patent Application No. 62/667,666, filed May 7, 2018;(11) U.S. Provisional Patent Application No. 62/667,679, filed May 7,2018; (12) U.S. Provisional Patent Application No. 62/667,691, filed May7, 2018; (13) U.S. Provisional Patent Application No. 62/667,699, filedMay 7, 2018; (14) U.S. Provisional Patent Application No. 62/667,717,filed May 7, 2018; (15) U.S. Provisional Patent Application No.62/667,749, filed May 7, 2018; (16) U.S. Provisional Patent ApplicationNo. 62/667,760, filed May 7, 2018; (17) U.S. Provisional PatentApplication No. 62/667,780, filed May 7, 2018; (18) U.S. ProvisionalPatent Application No. 62/667,792, filed May 7, 2018; (19) U.S.Provisional Patent Application No. 62/667,820, filed May 7, 2018; and(20) U.S. Provisional Patent Application No. 62/667,850, filed May 7,2018; which are commonly assigned herewith, and all of which is herebyincorporated herein by reference in their entireties with the same fullforce and effect as if set forth in their entirety herein (hereinafterreferred to as the “related applications”).

FIELD OF THE INVENTION

The present invention, in general, relates to configurable computingcircuitry, and more particularly, relates to a heterogeneous computingsystem which includes a self-scheduling processor, configurablecomputing circuitry with an embedded interconnection network, dynamicreconfiguration, and dynamic control over energy or power consumption.

BACKGROUND OF THE INVENTION

Many existing computing systems have reached significant limits forcomputation processing capabilities, both in terms of speed ofcomputation, energy (or power) consumption, and associated heatdissipation. For example, existing computing solutions have becomeincreasingly inadequate as the need for advanced computing technologiesgrows, such as to accommodate artificial intelligence and othersignificant computing applications.

Accordingly, there is an ongoing need for a computing architecturecapable of providing high performance and energy efficient solutions forcompute-intensive kernels, such as for computation of Fast FourierTransforms (FFTs) and finite impulse response (FIR) filters used insensing, communication, and analytic applications, such as syntheticaperture radar, 5G base stations, and graph analytic applications suchas graph clustering using spectral techniques, machine learning, 5Gnetworking algorithms, and large stencil codes, for example and withoutlimitation.

In addition, there is an ongoing need for a configurable computingarchitecture capable of being configured for any of these variousapplications, but most importantly, also capable of dynamicself-configuration and self-reconfiguration. Lastly, there is also anongoing need for a processor architecture capable of significantparallel processing and further interacting with and controlling aconfigurable computing architecture for performance of any of thesevarious applications.

SUMMARY OF THE INVENTION

As discussed in greater detail below, the representative apparatus,system and method provide for a computing architecture capable ofproviding high performance and energy efficient solutions forcompute-intensive kernels, such as for computation of Fast FourierTransforms (FFTs) and finite impulse response (FIR) filters used insensing, communication, and analytic applications, such as syntheticaperture radar, 5G base stations, and graph analytic applications suchas graph clustering using spectral techniques, machine learning, 5Gnetworking algorithms, and large stencil codes, for example and withoutlimitation.

Significantly, the various representative embodiments provide amulti-threaded, coarse-grained configurable computing architecturecapable of being configured for any of these various applications, butmost importantly, also capable of self-scheduling, dynamicself-configuration and self-reconfiguration, conditional branching,backpressure control for asynchronous signaling, ordered threadexecution and loop thread execution (including with data dependencies),automatically starting thread execution upon completion of datadependencies and/or ordering, providing loop access to privatevariables, providing rapid execution of loop threads using a reenterqueue, and using various thread identifiers for advanced loop execution,including nested loops.

Also as discussed in greater detail below, the representative apparatus,system and method provide for a processor architecture capable ofself-scheduling, significant parallel processing and further interactingwith and controlling a configurable computing architecture forperformance of any of these various applications.

In a representative embodiment, a system comprises: a first,interconnection network; a processor coupled to the interconnectionnetwork; a host interface coupled to the interconnection network; and atleast one configurable circuit cluster coupled to the interconnectionnetwork, the configurable circuit cluster comprising: a plurality ofconfigurable circuits arranged in an array; a second, asynchronouspacket network coupled to each configurable circuit of the plurality ofconfigurable circuits of the array; a third, synchronous network coupledto each configurable circuit of the plurality of configurable circuitsof the array; a memory interface circuit coupled to the asynchronouspacket network and to the interconnection network; and a dispatchinterface circuit coupled to the asynchronous packet network and to theinterconnection network.

For any of the various representative embodiments, the interconnectionnetwork may comprise: a first plurality of crossbar switches having aFolded Clos configuration and a plurality of direct, mesh connections atinterfaces with system endpoints 935. For any of the variousrepresentative embodiments, the asynchronous packet network maycomprise: a second plurality of crossbar switches, each crossbar switchcoupled to at least one configurable circuit of the plurality ofconfigurable circuits of the array and to another crossbar switch of thesecond plurality of crossbar switches. For any of the variousrepresentative embodiments, the synchronous network may comprise: aplurality of direct point-to-point connections coupling adjacentconfigurable circuits of the array of the plurality of configurablecircuits of the configurable circuit cluster.

In a representative embodiment, a configurable circuit may comprise: aconfigurable computation circuit; a plurality of synchronous networkinputs coupled to the configurable computation circuit; a plurality ofsynchronous network outputs coupled to the configurable computationcircuit; and a configuration memory coupled to the configurablecomputation circuit, to the control circuitry, to the synchronousnetwork inputs, and to the synchronous network outputs, with theconfiguration memory comprising: a first, instruction memory storing aplurality of data path configuration instructions to configure a datapath of the configurable computation circuit; and a second, instructionand instruction index memory storing a plurality of spoke instructionsand data path configuration instruction indices for selection of amaster synchronous input of the synchronous network inputs.

In a representative embodiment, each configurable circuit of theplurality of configurable circuits comprises: a configurable computationcircuit; a control circuit coupled to the configurable computationcircuit, the control circuit comprising: a memory control circuit; athread control circuit; and a plurality of control registers; a firstmemory circuit coupled to the configurable computation circuit; aplurality of synchronous network inputs coupled to the configurablecomputation circuit and to the synchronous network; a plurality ofsynchronous network outputs coupled to the configurable computationcircuit and to the synchronous network; an asynchronous network inputqueue coupled to the asynchronous packet network; an asynchronousnetwork output queue coupled to the asynchronous packet network; asecond, configuration memory circuit coupled to the configurablecomputation circuit, to the control circuitry, to the synchronousnetwork inputs, and to the synchronous network outputs, theconfiguration memory circuit comprising: a first, instruction memorystoring a plurality of data path configuration instructions to configurea data path of the configurable computation circuit; and a second,instruction and instruction index memory storing a plurality of spokeinstructions and data path configuration instruction indices forselection of a master synchronous input of the synchronous networkinputs.

In another representative embodiment, a system may comprise: a first,interconnection network; a processor coupled to the interconnectionnetwork; a host interface coupled to the interconnection network; and atleast one configurable circuit cluster coupled to the interconnectionnetwork, the configurable circuit cluster comprising: a plurality ofconfigurable circuits arranged in an array, each configurable circuitcomprising: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs and outputs coupled to the configurablecomputation circuit; an asynchronous network input queue and anasynchronous network output queue; a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs, the second, configuration memory comprising: a first,instruction memory storing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and a second, instruction and instruction index memory storing:a plurality of spoke instructions and data path configurationinstruction indices for selection of a master synchronous input of thesynchronous network inputs, for selection of a current data pathconfiguration instruction for the configurable computation circuit, andfor selection of a next data path instruction or next data pathinstruction index for a next configurable computation circuit; and acontrol circuit coupled to the configurable computation circuit, thecontrol circuit comprising: a memory control circuit; a plurality ofcontrol registers, wherein the plurality of control registers store acompletion table having a first, data completion count; and a threadcontrol circuit adapted to queue a thread for execution.

In another representative embodiment, a system may comprise: a first,interconnection network; a host interface coupled to the interconnectionnetwork; at least one configurable circuit cluster coupled to theinterconnection network, the configurable circuit cluster comprising aplurality of configurable circuits arranged in an array; and a processorcoupled to the interconnection network, the processor comprising: aprocessor core adapted to execute a plurality of instructions; and acore control circuit coupled to the processor core, the core controlcircuit comprising: an interconnection network interface coupleable toan interconnection network to receive a work descriptor data packet, todecode the received work descriptor data packet into an execution threadhaving an initial program count and any received argument; a threadcontrol memory coupled to the interconnection network interface andcomprising a plurality of registers, the plurality of registerscomprising a thread identifier pool register storing a plurality ofthread identifiers, a thread state register, a program count registerstoring the received program count, a data cache, and a general-purposeregister storing the received argument; an execution queue coupled tothe thread control memory; a control logic and thread selection circuitcoupled to the execution queue and to the thread control memory, thecontrol logic and thread selection circuit adapted to assign anavailable thread identifier to the execution thread, to place the threadidentifier in the execution queue, to select the thread identifier forexecution, to access the thread control memory using the threadidentifier as an index to select the initial program count for theexecution thread; and an instruction cache coupled to the processor coreand to the control logic and thread selection circuit to receive theinitial program count and provide to the processor core a correspondinginstruction for execution, of the plurality of instructions.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; and a configuration memorycoupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs, the configuration memory comprising: a first,instruction memory storing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and a second, instruction and instruction index memory storinga plurality of spoke instructions and data path configurationinstruction indices for selection of a current data path configurationinstruction for the configurable computation circuit.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; and a configuration memorycoupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs, the configuration memory comprising: a first,instruction memory storing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and a second, instruction and instruction index memory storinga plurality of spoke instructions and data path configurationinstruction indices for selection of a next data path configurationinstruction for a next configurable computation circuit.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a control circuit coupledto the configurable computation circuit; a first memory circuit coupledto the configurable computation circuit; a plurality of synchronousnetwork inputs coupled to the configurable computation circuit; aplurality of synchronous network outputs coupled to the configurablecomputation circuit; and a second, configuration memory circuit coupledto the configurable computation circuit, to the control circuitry, tothe synchronous network inputs, and to the synchronous network outputs,the configuration memory circuit comprising: a first, instruction memorystoring a plurality of data path configuration instructions to configurea data path of the configurable computation circuit; and a second,instruction and instruction index memory storing a plurality of spokeinstructions and data path configuration instruction indices forselection of a master synchronous input of the synchronous networkinputs.

In yet another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and a control circuit coupled to the configurablecomputation circuit, the control circuit comprising: a memory controlcircuit; a thread control circuit; and a plurality of control registers.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a configuration memorycoupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs, the configuration memory comprising: a first,instruction memory storing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and a second, instruction and instruction index memory storinga plurality of spoke instructions and data path configurationinstruction indices for selection of a next data path instruction ornext data path instruction index for a next configurable computationcircuit; and a conditional logic circuit coupled to the configurablecomputing circuit, wherein depending upon an output from theconfigurable computing circuit, the conditional logic circuit is adaptedto provide conditional branching by modifying the next data pathinstruction or next data path instruction index provided on a selectedoutput of the plurality of synchronous network outputs.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a control circuit coupledto the configurable computation circuit; a first memory circuit coupledto the configurable computation circuit; a plurality of synchronousnetwork inputs coupled to the configurable computation circuit; aplurality of synchronous network outputs coupled to the configurablecomputation circuit; an asynchronous network input queue coupled to anasynchronous packet network and to the first memory circuit; anasynchronous network output queue; and a flow control circuit coupled tothe asynchronous network output queue, the flow control circuit adaptedto generate a stop signal when a predetermined threshold has beenreached in the asynchronous network output queue.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and a control circuit coupled to the configurablecomputation circuit, the control circuit comprising: a memory controlcircuit; a thread control circuit; and a plurality of control registers,wherein the plurality of control registers store a loop table having aplurality of thread identifiers and, for each thread identifier, a nextthread identifier for execution following execution of a current threadto provide ordered thread execution.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and a control circuit coupled to the configurablecomputation circuit, the control circuit comprising: a memory controlcircuit; a plurality of control registers, wherein the plurality ofcontrol registers store a completion table having a first, datacompletion count; and a thread control circuit adapted to queue a threadfor execution when, for its thread identifier, its completion count hasdecremented to zero.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs and outputs coupled to the configurablecomputation circuit; an asynchronous network input queue and anasynchronous network output queue; a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs, the second, configuration memory comprising: a first,instruction memory storing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and a second, instruction and instruction index memory storing:a plurality of spoke instructions and data path configurationinstruction indices for selection of a master synchronous input of thesynchronous network inputs, for selection of a current data pathconfiguration instruction for the configurable computation circuit, andfor selection of a next data path instruction or next data pathinstruction index for a next configurable computation circuit; and theconfigurable circuit further comprising a control circuit coupled to theconfigurable computation circuit, the control circuit comprising: amemory control circuit; a plurality of control registers, wherein theplurality of control registers store a completion table having a first,data completion count; and a thread control circuit adapted to queue athread for execution when, for its thread identifier, its completioncount has decremented to zero.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and a control circuit coupled to the configurablecomputation circuit, the control circuit comprising: a memory controlcircuit; a plurality of control registers, wherein the plurality ofcontrol registers store a completion table having a first, datacompletion count; and a thread control circuit adapted to queue a threadfor execution when, for its thread identifier, its completion count hasdecremented to zero and its thread identifier is the next thread.

In yet another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and the configurable circuit further comprising acontrol circuit coupled to the configurable computation circuit, thecontrol circuit comprising: a memory control circuit; a thread controlcircuit; and a plurality of control registers storing a completion tablehaving a plurality of types of thread identifiers, with each type ofthread identifier indicating a loop level for loop and nested loopexecution, and wherein the plurality of control registers further storea top of thread identifiers stack to allow each type of threadidentifier access to private variables for a selected loop.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and a control circuit coupled to the configurablecomputation circuit, the control circuit comprising: a memory controlcircuit; a plurality of control registers; and a thread control circuitcomprising: a continuation queue storing one or more thread identifiersfor computation threads having completion counts allowing execution butdo not yet have an assigned thread identifier; and a reenter queuestoring one or more thread identifiers for computation threads havingcompletion counts allowing execution and having an assigned threadidentifier to provide for execution of the threads in the reenter queueupon a designated spoke count.

In another representative embodiment, a configurable circuit maycomprise: a configurable computation circuit; a first memory circuitcoupled to the configurable computation circuit; a plurality ofsynchronous network inputs coupled to the configurable computationcircuit; a plurality of synchronous network outputs coupled to theconfigurable computation circuit; and a second, configuration memorycircuit coupled to the configurable computation circuit, to the controlcircuitry, to the synchronous network inputs, and to the synchronousnetwork outputs; and a control circuit coupled to the configurablecomputation circuit, the control circuit comprising: a memory controlcircuit; a plurality of control registers storing a thread identifierpool and a completion table having a loop count of an active number ofloop threads; and a thread control circuit, wherein in response toreceipt of an asynchronous fabric message returning a thread identifierto the thread identifier pool, the control circuit decrements the loopcount and, when the loop count reaches zero, transmits an asynchronousfabric completion message.

In a representative embodiment, a system is disclosed, which maycomprise: an asynchronous packet network; a synchronous network; and aplurality of configurable circuits arranged in an array, eachconfigurable circuit of the plurality of configurable circuits coupledto both the synchronous network and to the asynchronous packet network,the plurality of configurable circuits adapted to perform a plurality ofcomputations using the synchronous network to form a plurality ofsynchronous domains, and the plurality of configurable circuits furtheradapted to generate and transmit a plurality of control messages overthe asynchronous packet network, the plurality of control messagescomprising one or more completion messages and continue messages.

In another representative embodiment, a system may comprise: a pluralityof configurable circuits arranged in an array; a synchronous networkcoupled to each configurable circuit of the plurality of configurablecircuits of the array; and an asynchronous packet network coupled toeach configurable circuit of the plurality of configurable circuits ofthe array.

In another representative embodiment, a system may comprise: aninterconnection network; a processor coupled to the interconnectionnetwork; and a plurality of configurable circuit clusters coupled to theinterconnection network.

In a representative embodiment, a system may comprise: aninterconnection network; a processor coupled to the interconnectionnetwork; a host interface coupled to the interconnection network; and aplurality of configurable circuit clusters coupled to theinterconnection network, each configurable circuit cluster of theplurality of configurable circuit clusters comprising: a plurality ofconfigurable circuits arranged in an array; a synchronous networkcoupled to each configurable circuit of the plurality of configurablecircuits of the array; an asynchronous packet network coupled to eachconfigurable circuit of the plurality of configurable circuits of thearray; a memory interface coupled to the asynchronous packet network andto the interconnection network; and a dispatch interface coupled to theasynchronous packet network and to the interconnection network.

In another representative embodiment, a system may comprise: ahierarchical interconnection network comprising a first plurality ofcrossbar switches having a Folded Clos configuration and a plurality ofdirect, mesh connections at interfaces with endpoints; a processorcoupled to the interconnection network; a host interface coupled to theinterconnection network; and a plurality of configurable circuitclusters coupled to the interconnection network, each configurablecircuit cluster of the plurality of configurable circuit clusterscomprising: a plurality of configurable circuits arranged in an array; asynchronous network coupled to each configurable circuit of theplurality of configurable circuits of the array and providing aplurality of direct connections between adjacent configurable circuitsof the array; an asynchronous packet network comprising a secondplurality of crossbar switches, each crossbar switch coupled to at leastone configurable circuit of the plurality of configurable circuits ofthe array and to another crossbar switch of the second plurality ofcrossbar switches; a memory interface coupled to the asynchronous packetnetwork and to the interconnection network; and a dispatch interfacecoupled to the asynchronous packet network and to the interconnectionnetwork.

In another representative embodiment, a system may comprise: aninterconnection network; a processor coupled to the interconnectionnetwork; a host interface coupled to the interconnection network; and aplurality of configurable circuit clusters coupled to theinterconnection network, each configurable circuit cluster of theplurality of configurable circuit clusters comprising: a synchronousnetwork; an asynchronous packet network; a memory interface coupled tothe asynchronous packet network and to the interconnection network; adispatch interface coupled to the asynchronous packet network and to theinterconnection network; and a plurality of configurable circuitsarranged in an array, each configurable circuit comprising: aconfigurable computation circuit; a control circuit coupled to theconfigurable computation circuit, the control circuit comprising: amemory control circuit; a thread control circuit; and a plurality ofcontrol registers; a first memory circuit coupled to the configurablecomputation circuit; a plurality of synchronous network inputs andoutputs coupled to the configurable computation circuit and to thesynchronous network; an asynchronous network input queue and anasynchronous network output queue coupled to the asynchronous packetnetwork; a second, configuration memory circuit coupled to theconfigurable computation circuit, to the control circuitry, to thesynchronous network inputs, and to the synchronous network outputs, theconfiguration memory circuit comprising: a first, instruction memorystoring a plurality of data path configuration instructions to configurea data path of the configurable computation circuit; and a second,instruction and instruction index memory storing a plurality of spokeinstructions and data path configuration instruction indices forselection of a master synchronous input of the synchronous networkinputs.

In any of the various representative embodiments, the second,instruction and instruction index memory may further store a pluralityof spoke instructions and data path configuration instruction indicesfor selection of a current data path configuration instruction for theconfigurable computation circuit.

In any of the various representative embodiments, the second,instruction and instruction index memory may further store a pluralityof spoke instructions and data path configuration instruction indicesfor selection of a next data path configuration instruction for a nextconfigurable computation circuit.

In any of the various representative embodiments, the second,instruction and instruction index memory may further store a pluralityof spoke instructions and data path configuration instruction indicesfor selection of a synchronous network output of the plurality ofsynchronous network outputs.

In any of the various representative embodiments, the configurablecircuit or system may further comprise: a configuration memorymultiplexer coupled to the first, instruction memory and to the second,instruction and instruction index memory.

In any of the various representative embodiments, when a selection inputof the configuration memory multiplexer has a first setting, the currentdata path configuration instruction may be selected using an instructionindex from the second, instruction and instruction index memory.

In any of the various representative embodiments, when the selectioninput of the configuration memory multiplexer has a second settingdifferent from the first setting, the current data path configurationinstruction may be selected using an instruction index from the mastersynchronous input.

In any of the various representative embodiments, the second,instruction and instruction index memory may further store a pluralityof spoke instructions and data path configuration instruction indicesfor configuration of portions of the configurable circuit independentlyfrom the current data path instruction.

In any of the various representative embodiments, a selected spokeinstruction and data path configuration instruction index of theplurality of spoke instructions and data path configuration instructionindices may be selected according to a modulo spoke count.

In any of the various representative embodiments, the configurablecircuit or system may further comprise: a conditional logic circuitcoupled to the configurable computing circuit.

In any of the various representative embodiments, depending upon anoutput from the configurable computing circuit, the conditional logiccircuit may be adapted to modify the next data path instruction indexprovided on a selected output of the plurality of synchronous networkoutputs.

In any of the various representative embodiments, depending upon anoutput from the configurable computing circuit, the conditional logiccircuit may be adapted to provide conditional branching by modifying thenext data path instruction or next data path instruction index providedon a selected output of the plurality of synchronous network outputs.

In any of the various representative embodiments, the conditional logiccircuit, when enabled, may be adapted to provide conditional branchingby ORing the least significant bit of the next data path instructionwith the output from the configurable computing circuit to designate thenext data path instruction or data path instruction index.

In any of the various representative embodiments, the conditional logiccircuit, when enabled, may be adapted to provide conditional branchingby ORing the least significant bit of the next data path instructionindex with the output from the configurable computing circuit todesignate the next data path instruction index.

In any of the various representative embodiments, the plurality ofsynchronous network inputs may comprise: a plurality of input registerscoupled to a plurality of communication lines of a synchronous network;and an input multiplexer coupled to the plurality of input registers andto the second, instruction and instruction index memory for selection ofthe master synchronous input.

In any of the various representative embodiments, the plurality ofsynchronous network outputs may comprise: a plurality of outputregisters coupled to a plurality of communication lines of thesynchronous network; and an output multiplexer coupled to theconfigurable computing circuit for selection of an output from theconfigurable computing circuit.

In any of the various representative embodiments, the configurablecircuit or system may further comprise: an asynchronous fabric statemachine coupled to the asynchronous network input queue and to theasynchronous network output queue, the asynchronous fabric state machineadapted to decode an input data packet received from the asynchronouspacket network and to assemble an output data packet for transmission onthe asynchronous packet network.

In any of the various representative embodiments, the asynchronouspacket network may comprise a plurality of crossbar switches, eachcrossbar switch coupled to a plurality of configurable circuits and toat least one other crossbar switch.

In any of the various representative embodiments, the configurablecircuit or system may further comprise: an array of a plurality ofconfigurable circuits, wherein: each configurable circuit is coupledthrough the plurality of synchronous network inputs and the plurality ofsynchronous network outputs to the synchronous network; and eachconfigurable circuit is coupled through the asynchronous network inputand the asynchronous network output to the asynchronous packet network.

In any of the various representative embodiments, the synchronousnetwork may comprise a plurality of direct point-to-point connectionscoupling adjacent configurable circuits of the array of the plurality ofconfigurable circuits.

In any of the various representative embodiments, each configurablecircuit may comprise: a direct, pass through connection between theplurality of input registers and the plurality of output registers. Inany of the various representative embodiments, the direct, pass throughconnection may provide a direct, point-to-point connection for datatransmission from a second configurable circuit received on thesynchronous network to a third configurable circuit transmitted on thesynchronous network.

In any of the various representative embodiments, the configurablecomputation circuit may comprise an arithmetic, logical and bitoperation circuit adapted to perform at least one integer operationselected from the group consisting of: signed and unsigned addition,absolute value, negate, logical NOT, add and negate, subtraction A-B,reverse subtraction B-A, signed and unsigned greater than, signed andunsigned greater than or equal to, signed and unsigned less than, signedand unsigned less than or equal to, comparison of equal or not equal to,logical AND operation, logical OR operation, logical XOR operation,logical NAND operation, logical NOR operation, logical NOT XORoperation, logical AND NOT operation, logical OR NOT operation, and aninterconversion between integer and floating point.

In any of the various representative embodiments, the configurablecomputation circuit may comprise an arithmetic, logical and bitoperation circuit adapted to perform at least one floating pointoperation selected from the group consisting of: signed and unsignedaddition, absolute value, negate, logical NOT, add and negate,subtraction A-B, reverse subtraction B-A, signed and unsigned greaterthan, signed and unsigned greater than or equal to, signed and unsignedless than, signed and unsigned less than or equal to, comparison ofequal or not equal to, logical AND operation, logical OR operation,logical XOR operation, logical NAND operation, logical NOR operation,logical NOT XOR operation, logical AND NOT operation, logical OR NOToperation, an interconversion between integer and floating point, andcombinations thereof.

In any of the various representative embodiments, the configurablecomputation circuit may comprise a multiply and shift operation circuitadapted to perform at least one integer operation selected from thegroup consisting of: multiply, shift, pass an input, signed and unsignedmultiply, signed and unsigned shift right, signed and unsigned shiftleft, bit order reversal, a permutation, an interconversion betweeninteger and floating point, and combinations thereof.

In any of the various representative embodiments, the configurablecomputation circuit may comprise a multiply and shift operation circuitadapted to perform at least floating point operation selected from thegroup consisting of: multiply, shift, pass an input, signed and unsignedmultiply, signed and unsigned shift right, signed and unsigned shiftleft, bit order reversal, a permutation, an interconversion betweeninteger and floating point, and combinations thereof.

In any of the various representative embodiments, the array of theplurality of configurable circuits may be further coupled to a firstinterconnection network. In any of the various representativeembodiments, the array of the plurality of configurable circuits mayfurther comprise: a third, system memory interface circuit; and adispatch interface circuit. In any of the various representativeembodiments, the dispatch interface circuit may be adapted to receive awork descriptor packet over the first interconnection network, and inresponse to the work descriptor packet, to generate one or more data andcontrol packets to the plurality of configurable circuits to configurethe plurality of configurable circuits for execution of a selectedcomputation.

In any of the various representative embodiments, the configurablecircuit or system may further comprise: a flow control circuit coupledto the asynchronous network output queue, the flow control circuitadapted to generate a stop signal when a predetermined threshold hasbeen reached in the asynchronous network output queue. In any of thevarious representative embodiments, in response to the stop signal, eachasynchronous network output queue stops outputting data packets on theasynchronous packet network. In any of the various representativeembodiments, in response to the stop signal, each configurablecomputation circuit stops executing upon completion of its currentinstruction.

In any of the various representative embodiments, a first plurality ofconfigurable circuits of the array of a plurality of configurablecircuits may be coupled in a first predetermined sequence through thesynchronous network to form a first synchronous domain; and wherein asecond plurality of configurable circuits of the array of a plurality ofconfigurable circuits are coupled in a second predetermined sequencethrough the synchronous network to form a second synchronous domain. Inany of the various representative embodiments, the first synchronousdomain may be adapted to generate a continuation message to the secondsynchronous domain transmitted through the asynchronous packet network.In any of the various representative embodiments, the second synchronousdomain may be adapted to generate a completion message to the firstsynchronous domain transmitted through the asynchronous packet network.

In any of the various representative embodiments, the plurality ofcontrol registers may store a completion table having a first, datacompletion count. In any of the various representative embodiments, theplurality of control registers further store the completion table havinga second, iteration count. In any of the various representativeembodiments, the plurality of control registers may further store a looptable having a plurality of thread identifiers and, for each threadidentifier, a next thread identifier for execution following executionof a current thread. In any of the various representative embodiments,the plurality of control registers may further store, in the loop table,an identification of a first iteration and an identification of a lastiteration.

In any of the various representative embodiments, the control circuitmay be adapted to queue a thread for execution when, for its threadidentifier, its completion count has decremented to zero and its threadidentifier is the next thread.

In any of the various representative embodiments, the control circuitmay be adapted to queue a thread for execution when, for its threadidentifier, its completion count indicates completion of any datadependencies. In any of the various representative embodiments, thecompletion count may indicate a predetermined number of completionmessages to be received, per selected thread of a plurality of threads,prior to execution of the selected thread.

In any of the various representative embodiments, the plurality ofcontrol registers may further store a completion table having aplurality of types of thread identifiers, with each type of threadidentifier indicating a loop level for loop and nested loop execution.

In any of the various representative embodiments, the plurality ofcontrol registers may further store a completion table having a loopcount of an active number of loop threads, and wherein in response toreceipt of an asynchronous fabric message returning a thread identifierto a thread identifier pool, the control circuit decrements the loopcount and, when the loop count reaches zero, transmits an asynchronousfabric completion message. In any of the various representativeembodiments, the plurality of control registers may further store a topof thread identifiers stack to allow each type of thread identifieraccess to private variables for a selected loop.

In any of the various representative embodiments, the control circuitmay further comprise: a continuation queue; and a reenter queue. In anyof the various representative embodiments, the continuation queue storesone or more thread identifiers for computation threads having completioncounts allowing execution but do not yet have an assigned threadidentifier. In any of the various representative embodiments, thereenter queue may store one or more thread identifiers for computationthreads having completion counts allowing execution and having anassigned thread identifier. In any of the various representativeembodiments, any thread having a thread identifier in the reenter queuemay be executed prior to execution of any thread having a threadidentifier in the continuation queue.

In any of the various representative embodiments, the control circuitmay further comprise: a priority queue, wherein any thread having athread identifier in the priority queue may be executed prior toexecution of any thread having a thread identifier in the continuationqueue or in the reenter queue.

In any of the various representative embodiments, the control circuitmay further comprise: a run queue, wherein any thread having a threadidentifier in the run queue may be executed upon an occurrence of aspoke count for the thread identifier.

In any of the various representative embodiments, the second,configuration memory circuit may comprise: a first, instruction memorystoring a plurality of data path configuration instructions to configurea data path of the configurable computation circuit; and a second,instruction and instruction index memory storing a plurality of spokeinstructions and data path configuration instruction indices forselection of a master synchronous input of the synchronous networkinputs.

In any of the various representative embodiments, the control circuitmay be adapted to self-schedule a computation thread for execution.

In any of the various representative embodiments, the conditional logiccircuit may be adapted to branch to a different, second next instructionfor execution by a next configurable circuit.

In any of the various representative embodiments, the control circuitmay be adapted to order computation threads for execution. In any of thevarious representative embodiments, the control circuit may be adaptedto order loop computation threads for execution. In any of the variousrepresentative embodiments, the control circuit may be adapted tocommence execution of computation threads in response to one or morecompletion signals from data dependencies.

Various method embodiments of configuring a configurable circuit arealso disclosed. A representative method embodiment may comprise: using afirst, instruction memory, providing a plurality of data pathconfiguration instructions to configure a data path of the configurablecomputation circuit; and using a second, instruction and instructionindex memory, providing a plurality of spoke instructions and data pathconfiguration instruction indices for selection of a master synchronousinput of a plurality of synchronous network inputs.

In any of the various representative embodiments, a method ofconfiguring a configurable circuit may comprise: using a first,instruction memory, providing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and using a second, instruction and instruction index memory,providing a plurality of spoke instructions and data path configurationinstruction indices for selection of a current data path configurationinstruction for the configurable computation circuit.

In any of the various representative embodiments, a method ofconfiguring a configurable circuit may comprise: using a first,instruction memory, providing a plurality of data path configurationinstructions to configure a data path of the configurable computationcircuit; and using a second, instruction and instruction index memory,providing a plurality of spoke instructions and data path configurationinstruction indices for selection of a next data path configurationinstruction for a next configurable computation circuit.

A method of controlling thread execution of a multi-threadedconfigurable circuit is also disclosed, with the configurable circuithaving a configurable computation circuit. A representative methodembodiment may comprise: using a conditional logic circuit, dependingupon an output from the configurable computing circuit, providingconditional branching by modifying the next data path instruction ornext data path instruction index provided to a next configurablecircuit.

Another representative method embodiment of controlling thread executionof a multi-threaded configurable circuit may comprise: using a flowcontrol circuit, generating a stop signal when a predetermined thresholdhas been reached in an asynchronous network output queue.

Another representative method embodiment of controlling thread executionof a multi-threaded configurable circuit may comprise: using a pluralityof control registers, storing a loop table having a plurality of threadidentifiers and, for each thread identifier, a next thread identifierfor execution following execution of a current thread to provide orderedthread execution.

Another representative method embodiment of controlling thread executionof a multi-threaded configurable circuit may comprise: using a pluralityof control registers, storing a completion table having a first, datacompletion count; and using a thread control circuit, queueing a threadfor execution when, for its thread identifier, its completion count hasdecremented to zero.

A method of configuring and controlling thread execution of amulti-threaded configurable circuit having a configurable computationcircuit is disclosed, with a representative method embodimentcomprising: using a first, instruction memory, providing a plurality ofconfiguration instructions to configure a data path of the configurablecomputation circuit; using a second, instruction and instruction indexmemory, providing a plurality of spoke instructions and data pathconfiguration instruction indices for selection of a master synchronousinput of a plurality of synchronous network inputs, for selection of acurrent data path configuration instruction for the configurablecomputation circuit, and for selection of a next data path instructionor next data path instruction index for a next configurable computationcircuit; using a plurality of control registers, providing a completiontable having a first, data completion count; and using a thread controlcircuit, queueing a thread for execution when, for its threadidentifier, its completion count has decremented to zero.

Another method of configuring and controlling thread execution of amulti-threaded configurable circuit may comprise: using a first,instruction memory, providing a plurality of configuration instructionsto configure a data path of the configurable computation circuit; usinga second, instruction and instruction index memory, providing aplurality of spoke instructions and data path configuration instructionindices for selection of a master synchronous input of a plurality ofsynchronous network inputs, for selection of a current data pathconfiguration instruction for the configurable computation circuit, andfor selection of a next data path instruction or next data pathinstruction index for a next configurable computation circuit; using aplurality of control registers, providing a completion table having afirst, data completion count; and using a thread control circuit,queueing a thread for execution when, for its thread identifier, itscompletion count has decremented to zero and its thread identifier isthe next thread.

Another method of controlling thread execution of a multi-threadedconfigurable circuit may comprise: using a plurality of controlregisters, storing a completion table having a plurality of types ofthread identifiers, with each type of thread identifier indicating aloop level for loop and nested loop execution, and wherein the pluralityof control registers further store a top of thread identifiers stack;and allowing each type of thread identifier access to private variablesfor a selected loop.

Another method of controlling thread execution of a multi-threadedconfigurable circuit may comprise: using a plurality of controlregisters, storing a completion table having a data completion count;using a thread control circuit, providing a continuation queue storingone or more thread identifiers for computation threads having completioncounts allowing execution but do not yet have an assigned threadidentifier; and using a thread control circuit, providing a reenterqueue storing one or more thread identifiers for computation threadshaving completion counts allowing execution and having an assignedthread identifier to provide for execution of the threads in the reenterqueue upon a designated spoke count.

Another method of controlling thread execution of a multi-threadedconfigurable circuit may comprise: using a plurality of controlregisters, storing a thread identifier pool and a completion tablehaving a loop count of an active number of loop threads; and using athread control circuit, in response to receipt of an asynchronous fabricmessage returning a thread identifier to the thread identifier pool,decrementing the loop count and, when the loop count reaches zero,transmitting an asynchronous fabric completion message.

In any of the various representative embodiments, the method may furthercomprise: using the second, instruction and instruction index memory,providing a plurality of spoke instructions and data path configurationinstruction indices for selection of a current data path configurationinstruction for the configurable computation circuit.

In any of the various representative embodiments, the method may furthercomprise: using the second, instruction and instruction index memory,providing a plurality of spoke instructions and data path configurationinstruction indices for selection of a next data path configurationinstruction for a next configurable computation circuit.

In any of the various representative embodiments, the method may furthercomprise: using the second, instruction and instruction index memory,providing a plurality of spoke instructions and data path configurationinstruction indices for selection of a synchronous network output of theplurality of synchronous network outputs.

In any of the various representative embodiments, the method may furthercomprise: using a configuration memory multiplexer, providing a firstselection setting to select the current data path configurationinstruction using an instruction index from the second, instruction andinstruction index memory.

In any of the various representative embodiments, the method may furthercomprise: using a configuration memory multiplexer, providing a secondselection setting, the second setting different from the first setting,to select the current data path configuration instruction using aninstruction index from a master synchronous input.

In any of the various representative embodiments, the method may furthercomprise: using the second, instruction and instruction index memory,providing a plurality of spoke instructions and data path configurationinstruction indices for configuration of portions of the configurablecircuit independently from the current data path instruction.

In any of the various representative embodiments, the method may furthercomprise: using a configuration memory multiplexer, selecting a spokeinstruction and data path configuration instruction index of theplurality of spoke instructions and data path configuration instructionindices according to a modulo spoke count.

In any of the various representative embodiments, the method may furthercomprise: using a conditional logic circuit and depending upon an outputfrom the configurable computing circuit, modifying the next data pathinstruction or next data path instruction index.

In any of the various representative embodiments, the method may furthercomprise: using a conditional logic circuit and depending upon an outputfrom the configurable computing circuit, providing conditional branchingby modifying the next data path instruction or next data pathinstruction index.

In any of the various representative embodiments, the method may furthercomprise: enabling a conditional logic circuit; and using theconditional logic circuit and depending upon an output from theconfigurable computing circuit, providing conditional branching by ORingthe least significant bit of the next data path instruction with theoutput from the configurable computing circuit to designate the nextdata path instruction or data path instruction index.

In any of the various representative embodiments, the method may furthercomprise: using an input multiplexer, selecting the master synchronousinput. In any of the various representative embodiments, the method mayfurther comprise: using an output multiplexer, selecting an output fromthe configurable computing circuit.

In any of the various representative embodiments, the method may furthercomprise: using an asynchronous fabric state machine coupled to anasynchronous network input queue and to an asynchronous network outputqueue, decoding an input data packet received from the asynchronouspacket network and assembling an output data packet for transmission onthe asynchronous packet network.

In any of the various representative embodiments, the method may furthercomprise: using the synchronous network, providing a plurality of directpoint-to-point connections coupling adjacent configurable circuits ofthe array of the plurality of configurable circuits.

In any of the various representative embodiments, the method may furthercomprise: using the configurable circuit, providing a direct, passthrough connection between a plurality of input registers and aplurality of output registers. In any of the various representativeembodiments, the direct, pass through connection provides a direct,point-to-point connection for data transmission from a secondconfigurable circuit received on the synchronous network to a thirdconfigurable circuit transmitted on the synchronous network.

In any of the various representative embodiments, the method may furthercomprise: using the configurable computation circuit, performing atleast one integer or floating point operation selected from the groupconsisting of: signed and unsigned addition, absolute value, negate,logical NOT, add and negate, subtraction A-B, reverse subtraction B-A,signed and unsigned greater than, signed and unsigned greater than orequal to, signed and unsigned less than, signed and unsigned less thanor equal to, comparison of equal or not equal to, logical AND operation,logical OR operation, logical XOR operation, logical NAND operation,logical NOR operation, logical NOT XOR operation, logical AND NOToperation, logical OR NOT operation, and an interconversion betweeninteger and floating point.

In any of the various representative embodiments, the method may furthercomprise: using the configurable computation circuit, performing atleast one integer or floating point operation selected from the groupconsisting of: multiply, shift, pass an input, signed and unsignedmultiply, signed and unsigned shift right, signed and unsigned shiftleft, bit order reversal, a permutation, an interconversion betweeninteger and floating point, and combinations thereof.

In any of the various representative embodiments, the method may furthercomprise: using a dispatch interface circuit, receiving a workdescriptor packet over the first interconnection network, and inresponse to the work descriptor packet, to generate one or more data andcontrol packets to the plurality of configurable circuits to configurethe plurality of configurable circuits for execution of a selectedcomputation.

In any of the various representative embodiments, the method may furthercomprise: using a flow control circuit, generating a stop signal when apredetermined threshold has been reached in the asynchronous networkoutput queue. In any of the various representative embodiments, inresponse to the stop signal, each asynchronous network output queuestops outputting data packets on the asynchronous packet network. In anyof the various representative embodiments, in response to the stopsignal, each configurable computation circuit stops executing uponcompletion of its current instruction.

In any of the various representative embodiments, the method may furthercomprise: coupling a first plurality of configurable circuits of thearray of a plurality of configurable circuits in a first predeterminedsequence through the synchronous network to form a first synchronousdomain; and coupling a second plurality of configurable circuits of thearray of a plurality of configurable circuits are coupled in a secondpredetermined sequence through the synchronous network to form a secondsynchronous domain.

In any of the various representative embodiments, the method may furthercomprise: generating a continuation message from the first synchronousdomain to the second synchronous domain for transmission through theasynchronous packet network.

In any of the various representative embodiments, the method may furthercomprise: generating a completion message from the second synchronousdomain to the first synchronous domain for transmission through theasynchronous packet network.

In any of the various representative embodiments, the method may furthercomprise storing a completion table having a first, data completioncount in the plurality of control registers.

In any of the various representative embodiments, the method may furthercomprise: storing the completion table having a second, iteration countin the plurality of control registers.

In any of the various representative embodiments, the method may furthercomprise: storing a loop table having a plurality of thread identifiersin the plurality of control registers and, for each thread identifier,storing a next thread identifier for execution following execution of acurrent thread.

In any of the various representative embodiments, the method may furthercomprise: storing in the loop table in the plurality of controlregisters, an identification of a first iteration and an identificationof a last iteration.

In any of the various representative embodiments, the method may furthercomprise: using the control circuit, queueing a thread for executionwhen, for its thread identifier, its completion count has decremented tozero.

In any of the various representative embodiments, the method may furthercomprise: using the control circuit, queueing a thread for executionwhen, for its thread identifier, its completion count has decremented tozero and its thread identifier is the next thread.

In any of the various representative embodiments, the method may furthercomprise: using the control circuit, queueing a thread for executionwhen, for its thread identifier, its completion count indicatescompletion of any data dependencies. In any of the variousrepresentative embodiments, the completion count may indicate apredetermined number of completion messages to be received, per selectedthread of a plurality of threads, prior to execution of the selectedthread.

In any of the various representative embodiments, the method may furthercomprise: storing a completion table, in the plurality of controlregisters, having a plurality of types of thread identifiers, with eachtype of thread identifier indicating a loop level for loop and nestedloop execution.

In any of the various representative embodiments, the method may furthercomprise: storing, in the plurality of control registers, a completiontable having a loop count of an active number of loop threads, andwherein in response to receipt of an asynchronous fabric messagereturning a thread identifier to a thread identifier pool, using thecontrol circuit, decrementing the loop count and, when the loop countreaches zero, transmitting an asynchronous fabric completion message.

In any of the various representative embodiments, the method may furthercomprise: storing a top of thread identifiers stack in the plurality ofcontrol registers to allow each type of thread identifier access toprivate variables for a selected loop.

In any of the various representative embodiments, the method may furthercomprise: using a continuation queue, storing one or more threadidentifiers for computation threads having completion counts allowingexecution but do not yet have an assigned thread identifier.

In any of the various representative embodiments, the method may furthercomprise: using a reenter queue, storing one or more thread identifiersfor computation threads having completion counts allowing execution andhaving an assigned thread identifier.

In any of the various representative embodiments, the method may furthercomprise: executing any thread having a thread identifier in the reenterqueue prior to execution of any thread having a thread identifier in thecontinuation queue.

In any of the various representative embodiments, the method may furthercomprise: executing any thread having a thread identifier in a priorityqueue prior to execution of any thread having a thread identifier in thecontinuation queue or in the reenter queue.

In any of the various representative embodiments, the method may furthercomprise: executing any thread in a run queue upon an occurrence of aspoke count for the thread identifier.

In any of the various representative embodiments, the method may furthercomprise: using a control circuit, self-scheduling a computation threadfor execution.

In any of the various representative embodiments, the method may furthercomprise: using the conditional logic circuit, branching to a different,second next instruction for execution by a next configurable circuit.

In any of the various representative embodiments, the method may furthercomprise: using the control circuit, ordering computation threads forexecution.

In any of the various representative embodiments, the method may furthercomprise: using the control circuit, ordering loop computation threadsfor execution.

In any of the various representative embodiments, the method may furthercomprise: using the control circuit, commencing execution of computationthreads in response to one or more completion signals from datadependencies.

A self-scheduling processor is disclosed. In a representativeembodiment, the processor comprises: a processor core adapted to executea received instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to automaticallyschedule an instruction for execution by the processor core in responseto a received work descriptor data packet. In another representativeembodiment, the processor comprises: a processor core adapted to executea received instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to automaticallyschedule an instruction for execution by the processor core in responseto a received event data packet.

A multi-threaded, self-scheduling processor is also disclosed which cancreate threads on local or remote compute elements. In a representativeembodiment, the processor comprises: a processor core adapted to executea fiber create instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to automaticallyschedule the fiber create instruction for execution by the processorcore and generate one or more work descriptor data packets to anotherprocessor or hybrid threading fabric circuit for execution of acorresponding plurality of execution threads. In another representativeembodiment, the processor comprises: a processor core adapted to executea fiber create instruction; and a core control circuit coupled to theprocessor core, the core control circuit adapted to schedule the fibercreate instruction for execution by the processor core, to reserve apredetermined amount of memory space in a thread control memory to storereturn arguments, and to generate one or more work descriptor datapackets to another processor or hybrid threading fabric circuit forexecution of a corresponding plurality of execution threads.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interface; athread control memory coupled to the interconnection network interface;an execution queue coupled to the thread control memory; a control logicand thread selection circuit coupled to the execution queue, to thethread control memory; and an instruction cache coupled to the controllogic and thread selection circuit; and further, a processor core iscoupled to the instruction cache of the core control circuit.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interface; athread control memory coupled to the interconnection network interface;a network response memory; an execution queue coupled to the threadcontrol memory; a control logic and thread selection circuit coupled tothe execution queue, to the thread control memory; an instruction cachecoupled to the control logic and thread selection circuit; and a commandqueue; and further, a processor core is coupled to the instruction cacheand to the command queue of the core control circuit.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; an execution queue coupled to the thread control memory; and acontrol logic and thread selection circuit coupled to the executionqueue, the control logic and thread selection circuit adapted to assignan available thread identifier to the execution thread, to automaticallyplace the thread identifier in the execution queue, and to periodicallyselect the thread identifier for execution of the execution thread.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; an execution queue coupled to the thread control memory; and acontrol logic and thread selection circuit coupled to the executionqueue, the control logic and thread selection circuit adapted to assignan available thread identifier to the execution thread, to automaticallyplace the thread identifier in the execution queue, and to periodicallyselect the thread identifier for execution of an instruction of anexecution thread by a processor core.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: an execution queue coupled to thethread control memory; and a control logic and thread selection circuitcoupled to the execution queue, the control logic and thread selectioncircuit adapted to assign an available thread identifier to theexecution thread, to automatically place the thread identifier in theexecution queue, and to periodically select the thread identifier forexecution of an instruction of an execution thread by the processorcore.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, aprogram count register storing a received program count, a data cache,and a general-purpose register storing a received argument; an executionqueue coupled to the thread control memory; and a control logic andthread selection circuit coupled to the execution queue, the controllogic and thread selection circuit adapted to assign an available threadidentifier to the execution thread, to automatically place the threadidentifier in the execution queue, and to periodically select the threadidentifier for execution of an instruction of the execution thread bythe processor core, the processor core using data stored in the datacache or general-purpose register.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, aprogram count register storing a received program count, and threadstate registers storing a valid state or a paused state for each threadidentifier of the plurality of thread identifiers; an execution queuecoupled to the thread control memory; and a control logic and threadselection circuit coupled to the execution queue, the control logic andthread selection circuit adapted to assign an available threadidentifier to the execution thread, to automatically place the threadidentifier in the execution queue when it has a valid state, and for aslong as the valid state remains, to periodically select the threadidentifier for execution of an instruction of the execution thread bythe processor core until completion of the execution thread.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, aprogram count register storing a received program count, and threadstate registers storing a valid state or a paused state for each threadidentifier of the plurality of thread identifiers; an execution queuecoupled to the thread control memory; and a control logic and threadselection circuit coupled to the execution queue, the control logic andthread selection circuit adapted to assign an available threadidentifier to the execution thread, to automatically place the threadidentifier in the execution queue when it has a valid state, and for aslong as the valid state remains, to periodically select the threadidentifier for execution of an instruction of the execution thread bythe processor core, and to pause thread execution by not returning thethread identifier to the execution queue when it has a pause state.

In another representative embodiment, a processor comprises: a processorcore and a core control circuit coupled to the processor core, with thecore control circuit comprising: a thread control memory comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, athread state register, a program count register storing a receivedprogram count, a data cache, and a general-purpose register storing areceived argument; an execution queue coupled to the thread controlmemory; and a control logic and thread selection circuit coupled to theexecution queue, the control logic and thread selection circuit adaptedto assign an available thread identifier to the execution thread, toautomatically place the thread identifier in the execution queue, and toperiodically select the thread identifier for execution of aninstruction of an execution thread by the processor core.

In another representative embodiment, a processor comprises: a processorcore adapted to execute a plurality of instructions; and a core controlcircuit coupled to the processor core, with the core control circuitcomprising: an interconnection network interface coupleable to aninterconnection network to receive a work descriptor data packet, todecode the received work descriptor data packet into an execution threadhaving an initial program count and any received argument; a threadcontrol memory coupled to the interconnection network interface andcomprising a plurality of registers, the plurality of registerscomprising a thread identifier pool register storing a plurality ofthread identifiers, a thread state register, a program count registerstoring the received program count, a data cache, and a general-purposeregister storing the received argument; an execution queue coupled tothe thread control memory; a control logic and thread selection circuitcoupled to the execution queue and to the thread control memory, thecontrol logic and thread selection circuit adapted to assign anavailable thread identifier to the execution thread, to place the threadidentifier in the execution queue, to select the thread identifier forexecution, to access the thread control memory using the threadidentifier as an index to select the initial program count for theexecution thread; and an instruction cache coupled to the processor coreand to the control logic and thread selection circuit to receive theinitial program count and provide to the processor core a correspondinginstruction for execution, of the plurality of instructions.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; a thread control memory coupled to the interconnection networkinterface and comprising a plurality of registers, the plurality ofregisters comprising a thread identifier pool register storing aplurality of thread identifiers, a thread state register, a programcount register storing the received program count, a data cache, and ageneral-purpose register storing the received argument; an executionqueue coupled to the thread control memory; a control logic and threadselection circuit coupled to the execution queue and to the threadcontrol memory, the control logic and thread selection circuit adaptedto assign an available thread identifier to the execution thread, toautomatically place the thread identifier in the execution queue, toperiodically select the thread identifier for execution, to access thethread control memory using the thread identifier as an index to selectthe initial program count for the execution thread; and an instructioncache coupled to the control logic and thread selection circuit toreceive the initial program count and provide a correspondinginstruction for execution; and further, a processor core is coupled tothe instruction cache of the core control circuit, the processor coreadapted to execute the corresponding instruction.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a work descriptordata packet, to decode the received work descriptor data packet into anexecution thread having an initial program count and any receivedargument; a thread control memory coupled to the interconnection networkinterface and comprising a plurality of registers, the plurality ofregisters comprising a thread identifier pool register storing aplurality of thread identifiers, a thread state register, a programcount register storing the received program count, and a general-purposeregister storing the received argument; an execution queue coupled tothe thread control memory; a control logic and thread selection circuitcoupled to the execution queue and to the thread control memory, thecontrol logic and thread selection circuit adapted to assign anavailable thread identifier to the execution thread, to place the threadidentifier in the execution queue, to select the thread identifier forexecution, to access the thread control memory using the threadidentifier as an index to select the initial program count for theexecution thread; an instruction cache coupled to the control logic andthread selection circuit to receive the initial program count andprovide a corresponding instruction for execution; and a command queue;and further, a processor core is coupled to the instruction cache andthe command queue of the core control circuit, the processor coreadapted to execute the corresponding instruction.

In another representative embodiment, a processor comprises: a corecontrol circuit coupled to the interconnection network interface andcomprising: an interconnection network interface coupleable to aninterconnection network to receive a work descriptor data packet, todecode the received work descriptor data packet into an execution threadhaving an initial program count and any received argument; a threadcontrol memory coupled to the interconnection network interface andcomprising a plurality of registers, the plurality of registerscomprising a thread identifier pool register storing a plurality ofthread identifiers, a thread state register, a program count registerstoring the received program count, and a general-purpose registerstoring the received argument; an execution queue coupled to the threadcontrol memory; a control logic and thread selection circuit coupled tothe execution queue and to the thread control memory, the control logicand thread selection circuit adapted to assign an available threadidentifier to the execution thread, to place the thread identifier inthe execution queue, to select the thread identifier for execution, toaccess the thread control memory using the thread identifier as an indexto select the initial program count for the execution thread, and aninstruction cache coupled to the control logic and thread selectioncircuit to receive the initial program count and provide a correspondinginstruction for execution; and further, a processor core is coupled tothe instruction cache of the core control circuit, the processor coreadapted to execute the corresponding instruction.

In another representative embodiment, a processor comprises: a corecontrol circuit comprising: an interconnection network interfacecoupleable to an interconnection network to receive a call workdescriptor data packet, to decode the received work descriptor datapacket into an execution thread having an initial program count and anyreceived argument, and to encode a work descriptor packet fortransmission to other processing elements; a thread control memorycoupled to the interconnection network interface and comprising aplurality of registers, the plurality of registers comprising a threadidentifier pool register storing a plurality of thread identifiers, athread state register, a program count register storing the receivedprogram count, and a general-purpose register storing the receivedargument; an execution queue coupled to the thread control memory; anetwork response memory coupled to the interconnection networkinterface; a control logic and thread selection circuit coupled to theexecution queue, to the thread control memory, and to the instructioncache, the control logic and thread selection circuit adapted to assignan available thread identifier to the execution thread, to place thethread identifier in the execution queue, to select the threadidentifier for execution, to access the thread control memory using thethread identifier as an index to select the initial program count forthe execution thread; an instruction cache coupled to the control logicand thread selection circuit to receive the initial program count andprovide a corresponding instruction for execution; and a command queuestoring one or more commands for generation of one or more workdescriptor packets; and further, a processor core is coupled to theinstruction cache and the command queue of the core control circuit, theprocessor core adapted to execute the corresponding instruction.

For any of the various representative embodiments, the core controlcircuit may further comprise: an interconnection network interfacecoupleable to an interconnection network, the interconnection networkinterface adapted to receive a work descriptor data packet, to decodethe received work descriptor data packet into an execution thread havingan initial program count and any received argument. For any of thevarious representative embodiments, the interconnection networkinterface may be further adapted to receive an event data packet, todecode the received event data packet into an event identifier and anyreceived argument.

For any of the various representative embodiments, the core controlcircuit may further comprise: a control logic and thread selectioncircuit coupled to the interconnection network interface, the controllogic and thread selection circuit adapted to assign an available threadidentifier to the execution thread.

For any of the various representative embodiments, the core controlcircuit may further comprise: a thread control memory having a pluralityof registers, with the plurality of registers comprising one or more ofthe following, in any selected combination: a thread identifier poolregister storing a plurality of thread identifiers; a thread stateregister; a program count register storing a received initial programcount; a general-purpose register storing the received argument; apending fiber return count register; a return argument buffer orregister; a return argument link list register; a custom atomictransaction identifier register; an event state register; an event maskregister; and a data cache.

For any of the various representative embodiments, the interconnectionnetwork interface may be further adapted to store the execution threadhaving the initial program count and any received argument in the threadcontrol memory using a thread identifier as an index to the threadcontrol memory.

For any of the various representative embodiments, the core controlcircuit may further comprise: a control logic and thread selectioncircuit coupled to the thread control memory and to the interconnectionnetwork interface, the control logic and thread selection circuitadapted to assign an available thread identifier to the executionthread.

For any of the various representative embodiments, the core controlcircuit may further comprise: an execution queue coupled to the threadcontrol memory, the execution queue storing one or more threadidentifiers.

For any of the various representative embodiments, the core controlcircuit may further comprise: a control logic and thread selectioncircuit coupled to the execution queue, to the interconnection networkinterface, and to the thread control memory, the control logic andthread selection circuit adapted to assign an available threadidentifier to the execution thread, to place the thread identifier inthe execution queue, to select the thread identifier for execution, andto access the thread control memory using the thread identifier as anindex to select the initial program count for the execution thread.

For any of the various representative embodiments, the core controlcircuit may further comprise: an instruction cache coupled to thecontrol logic and thread selection circuit to receive the initialprogram count and provide a corresponding instruction for execution.

In another representative embodiment, the processor further may furthercomprise: a processor core coupled to the instruction cache of the corecontrol circuit, the processor core adapted to execute the correspondinginstruction.

For any of the various representative embodiments, the core controlcircuit may be further adapted to assign an initial valid state to theexecution thread. For any of the various representative embodiments, thecore control circuit may be further adapted to assign a pause state tothe execution thread in response to the processor core executing amemory load instruction. For any of the various representativeembodiments, the core control circuit may be further adapted to assign apause state to the execution thread in response to the processor coreexecuting a memory store instruction.

For any of the various representative embodiments, the core controlcircuit may be further adapted to end execution of a selected thread inresponse to the execution of a return instruction by the processor core.For any of the various representative embodiments, the core controlcircuit may be further adapted to return a corresponding threadidentifier of the selected thread to the thread identifier pool registerin response to the execution of a return instruction by the processorcore. For any of the various representative embodiments, the corecontrol circuit may be further adapted to clear the registers of thethread control memory indexed by the corresponding thread identifier ofthe selected thread in response to the execution of a return instructionby the processor core.

For any of the various representative embodiments, the interconnectionnetwork interface may be further adapted to generate a return workdescriptor packet in response to the execution of a return instructionby the processor core.

For any of the various representative embodiments, the core controlcircuit may further comprise: a network response memory. For any of thevarious representative embodiments, the network response memory maycomprise one or more of the following, in any selected combination: amemory request register; a thread identifier and transaction identifierregister; a request cache line index register; a bytes register; and ageneral-purpose register index and type register.

For any of the various representative embodiments, the interconnectionnetwork interface may be adapted to generate a point-to-point event datamessage. For any of the various representative embodiments, theinterconnection network interface may be adapted to generate a broadcastevent data message.

For any of the various representative embodiments, the core controlcircuit may be further adapted to use an event mask stored in the eventmask register to respond to a received event data packet. For any of thevarious representative embodiments, the core control circuit may befurther adapted to determine an event number corresponding to a receivedevent data packet. For any of the various representative embodiments,the core control circuit may be further adapted to change the status ofa thread identifier from pause to valid in response to a received eventdata packet to resume execution of a corresponding execution thread. Forany of the various representative embodiments, the core control circuitmay be further adapted to change the status of a thread identifier frompause to valid in response to an event number of a received event datapacket to resume execution of a corresponding execution thread.

For any of the various representative embodiments, the control logic andthread selection circuit may be further adapted to successively select anext thread identifier from the execution queue for execution of asingle instruction of a corresponding execution thread. For any of thevarious representative embodiments, the control logic and threadselection circuit may be further adapted to perform a round-robinselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread. For any of the variousrepresentative embodiments, the control logic and thread selectioncircuit may be further adapted to perform a round-robin selection of anext thread identifier from the execution queue, of the plurality ofthread identifiers, each for execution of a single instruction of acorresponding execution thread until completion of the execution thread.For any of the various representative embodiments, the control logic andthread selection circuit may be further adapted to perform a barrelselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread.

For any of the various representative embodiments, the control logic andthread selection circuit may be further adapted to assign a valid statusor a pause status to a thread identifier. For any of the variousrepresentative embodiments, the control logic and thread selectioncircuit may be further adapted to assign a priority status to a threadidentifier. For any of the various representative embodiments, thecontrol logic and thread selection circuit may be further adapted to,following execution of a corresponding instruction, to return thecorresponding thread identifier to the execution queue with an assignedvalid status and an assigned priority.

For any of the various representative embodiments, the core controlcircuit may further comprise: a network command queue coupled to theprocessor core.

For any of the various representative embodiments, the interconnectionnetwork interface may comprise: an input queue; a packet decoder circuitcoupled to the input queue, to the control logic and thread selectioncircuit, and to the thread control memory; an output queue; and a packetencoder circuit coupled to the output queue, to the network responsememory, and to the network command queue.

For any of the various representative embodiments, the execution queuemay further comprise: a first priority queue; and a second priorityqueue. For any of the various representative embodiments, the controllogic and thread selection circuit may further comprise: threadselection control circuitry coupled to the execution queue, the threadselection control circuitry adapted to select a thread identifier fromthe first priority queue at a first frequency and to select a threadidentifier from the second priority queue at a second frequency, thesecond frequency lower than the first frequency. For any of the variousrepresentative embodiments, the thread selection control circuitry maybe adapted to determine the second frequency as a skip count fromselection of a thread identifier from the first priority queue.

For any of the various representative embodiments, the core controlcircuit may further comprise: data path control circuitry adapted tocontrol access size over the first interconnection network. For any ofthe various representative embodiments, the core control circuit mayfurther comprise: data path control circuitry adapted to increase ordecrease memory load access size in response to time-averaged usagelevels. For any of the various representative embodiments, the corecontrol circuit may further comprise: data path control circuitryadapted to increase or decrease memory store access size in response totime-averaged usage levels. For any of the various representativeembodiments, the control logic and thread selection circuit may befurther adapted to increase a size of a memory load access request tocorrespond to a cache line boundary of the data cache.

For any of the various representative embodiments, the core controlcircuit may further comprise: system call circuitry adapted to generateone or more system calls to a host processor. For any of the variousrepresentative embodiments, the system call circuitry may furthercomprise: a plurality of system call credit registers storing apredetermined credit count to modulate a number of system calls in anypredetermined period of time.

For any of the various representative embodiments, the core controlcircuit may be further adapted, in response to a request from a hostprocessor, to generate a command to the command queue for theinterconnection network interface to copy and transmit all data from thethread control memory corresponding to a selected thread identifier formonitoring thread state.

For any of the various representative embodiments, the processor coremay be adapted to execute a fiber create instruction to generate one ormore commands to the command queue for the interconnection networkinterface to generate one or more call work descriptor packets toanother processor core or to a hybrid threading fabric circuit. For anyof the various representative embodiments, the core control circuit maybe further adapted, in response to execution of a fiber createinstruction by the processor core, to reserve a predetermined amount ofmemory space in the general-purpose registers or return argumentregisters.

For any of the various representative embodiments, in response to thegeneration of one or more call work descriptor packets to anotherprocessor core or to a hybrid threading fabric, the core control circuitmay be adapted to store a thread return count in the thread returnregister. For any of the various representative embodiments, in responseto receipt of a return data packet, the core control circuit may beadapted to decrement the thread return count stored in the thread returnregister. For any of the various representative embodiments, in responseto the thread return count in the thread return register beingdecremented to zero, the core control circuit may be adapted to change apaused status to a valid status for a corresponding thread identifierfor subsequent execution of a thread return instruction for completionof the created fibers or threads.

For any of the various representative embodiments, the processor coremay be adapted to execute a waiting or nonwaiting fiber joininstruction. For any of the various representative embodiments, theprocessor core may be adapted to execute a fiber join all instruction.

For any of the various representative embodiments, the processor coremay be adapted to execute a non-cached read or load instruction todesignate a general-purpose register for storage of data received from amemory. For any of the various representative embodiments, the processorcore may be adapted to execute a non-cached write or store instructionto designate data in a general-purpose register for storage in a memory.

For any of the various representative embodiments, the core controlcircuit may be adapted to assign a transaction identifier to any load orstore request to memory and to correlate the transaction identifier witha thread identifier.

For any of the various representative embodiments, the processor coremay be adapted to execute a first thread priority instruction to assigna first priority to an execution thread having a corresponding threadidentifier. For any of the various representative embodiments, theprocessor core may be adapted to execute a second thread priorityinstruction to assign a second priority to an execution thread having acorresponding thread identifier.

For any of the various representative embodiments, the processor coremay be adapted to execute a custom atomic return instruction to completean executing thread of a custom atomic operation. For any of the variousrepresentative embodiments, in conjunction with a memory controller, theprocessor core may be adapted to execute a floating point atomic memoryoperation. For any of the various representative embodiments, inconjunction with a memory controller, the processor core may be adaptedto execute a custom atomic memory operation.

A method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; and automatically scheduling theinstruction for execution in response to the received work descriptordata packet.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivingan event data packet; and automatically scheduling the instruction forexecution in response to the received event data packet.

A method of a first processing element to generate a plurality ofexecution threads for performance by a second processing element is alsodisclosed, with a representative method embodiment comprising: executinga fiber create instruction; and in response to the execution of thefiber create instruction generating one or more work descriptor datapackets to the second processing element for execution of the pluralityof execution threads.

A method of a first processing element to generate a plurality ofexecution threads for performance by a second processing element is alsodisclosed, with a representative method embodiment comprising: executinga fiber create instruction; and in response to the execution of thefiber create instruction reserving a predetermined amount of memoryspace in a thread control memory to store return arguments andgenerating one or more work descriptor data packets to the secondprocessing element for execution of the plurality of execution threads.

A method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; assigning an available thread identifier to theexecution thread; automatically queuing the thread identifier forexecution of the execution thread; and periodically selecting the threadidentifier for execution of the execution thread.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; assigning an available thread identifier to theexecution thread; automatically queuing the thread identifier forexecution of the execution thread when it has a valid state; and for aslong as the valid state remains, periodically selecting the threadidentifier for execution of an instruction of the execution thread untilcompletion of the execution thread.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; assigning an available thread identifier to theexecution thread; automatically queuing the thread identifier in anexecution queue for execution of the execution thread when it has avalid state; and for as long as the valid state remains, periodicallyselecting the thread identifier for execution of an instruction of theexecution thread; and pausing thread execution by not returning thethread identifier to the execution queue when it has a pause state.

Another method of self-scheduling execution of an instruction is alsodisclosed, with a representative method embodiment comprising: receivinga work descriptor data packet; decoding the received work descriptordata packet into an execution thread having an initial program count andany received argument; storing the initial program count and anyreceived argument in a thread control memory; assigning an availablethread identifier to the execution thread; automatically queuing thethread identifier for execution of the execution thread when it has avalid state; accessing the thread control memory using the threadidentifier as an index to select the initial program count for theexecution thread; and for as long as the valid state remains,periodically selecting the thread identifier for execution of aninstruction of the execution thread until completion of the executionthread.

For any of the various representative embodiments, the method mayfurther comprise: receiving an event data packet; and decoding thereceived event data packet into an event identifier and any receivedargument.

For any of the various representative embodiments, the method mayfurther comprise: assigning an initial valid state to the executionthread.

For any of the various representative embodiments, the method mayfurther comprise: assigning a pause state to the execution thread inresponse to the execution of a memory load instruction. For any of thevarious representative embodiments, the method may further comprise:assigning a pause state to the execution thread in response to theexecution of a memory store instruction.

For any of the various representative embodiments, the method mayfurther comprise: terminating execution of a selected thread in responseto the execution of a return instruction. For any of the variousrepresentative embodiments, the method may further comprise: returning acorresponding thread identifier of the selected thread to the threadidentifier pool in response to the execution of a return instruction.For any of the various representative embodiments, the method mayfurther comprise: clearing the registers of a thread control memoryindexed by the corresponding thread identifier of the selected thread inresponse to the execution of a return instruction. For any of thevarious representative embodiments, the method may further comprise:generating a return work descriptor packet in response to the executionof a return instruction.

For any of the various representative embodiments, the method mayfurther comprise: generating a point-to-point event data message. Forany of the various representative embodiments, the method may furthercomprise: generating a broadcast event data message.

For any of the various representative embodiments, the method mayfurther comprise: using an event mask to respond to a received eventdata packet. For any of the various representative embodiments, themethod may further comprise: determining an event number correspondingto a received event data packet. For any of the various representativeembodiments, the method may further comprise: changing the status of athread identifier from pause to valid in response to a received eventdata packet to resume execution of a corresponding execution thread. Forany of the various representative embodiments, the method may furthercomprise: changing the status of a thread identifier from pause to validin response to an event number of a received event data packet to resumeexecution of a corresponding execution thread.

For any of the various representative embodiments, the method mayfurther comprise: successively selecting a next thread identifier fromthe execution queue for execution of a single instruction of acorresponding execution thread. For any of the various representativeembodiments, the method may further comprise: performing a round-robinselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread. For any of the variousrepresentative embodiments, the method may further comprise: performinga round-robin selection of a next thread identifier from the executionqueue, of the plurality of thread identifiers, each for execution of asingle instruction of a corresponding execution thread until completionof the execution thread. For any of the various representativeembodiments, the method may further comprise: performing a barrelselection of a next thread identifier from the execution queue, of theplurality of thread identifiers, each for execution of a singleinstruction of a corresponding execution thread.

For any of the various representative embodiments, the method mayfurther comprise: assigning a valid status or a pause status to a threadidentifier. For any of the various representative embodiments, themethod may further comprise: assigning a priority status to a threadidentifier.

For any of the various representative embodiments, the method mayfurther comprise: following execution of a corresponding instruction,returning the corresponding thread identifier to the execution queuewith an assigned valid status and an assigned priority.

For any of the various representative embodiments, the method mayfurther comprise: selecting a thread identifier from a first priorityqueue at a first frequency and selecting a thread identifier from asecond priority queue at a second frequency, the second frequency lowerthan the first frequency. For any of the various representativeembodiments, the method may further comprise: determining the secondfrequency as a skip count from selection of a thread identifier from thefirst priority queue.

For any of the various representative embodiments, the method mayfurther comprise: controlling data path access size. For any of thevarious representative embodiments, the method may further comprise:increasing or decreasing memory load access size in response totime-averaged usage levels. For any of the various representativeembodiments, the method may further comprise: increasing or decreasingmemory store access size in response to time-averaged usage levels. Forany of the various representative embodiments, the method may furthercomprise: increasing a size of a memory load access request tocorrespond to a cache line boundary of the data cache.

For any of the various representative embodiments, the method mayfurther comprise: generating one or more system calls to a hostprocessor. For any of the various representative embodiments, the methodmay further comprise: using a predetermined credit count, modulating anumber of system calls in any predetermined period of time.

For any of the various representative embodiments, the method mayfurther comprise: in response to a request from a host processor,copying and transmitting all data from a thread control memorycorresponding to a selected thread identifier for monitoring threadstate.

For any of the various representative embodiments, the method mayfurther comprise: executing a fiber create instruction to generate oneor more commands to generate one or more call work descriptor packets toanother processor core or to a hybrid threading fabric circuit. For anyof the various representative embodiments, the method may furthercomprise: in response to execution of a fiber create instruction,reserving a predetermined amount of memory space for storing any returnarguments. For any of the various representative embodiments, the methodmay further comprise: in response to the generation of one or more callwork descriptor packets, storing a thread return count in the threadreturn register. For any of the various representative embodiments, themethod may further comprise: in response to receipt of a return datapacket, decrementing the thread return count stored in the thread returnregister. For any of the various representative embodiments, the methodmay further comprise: in response to the thread return count in thethread return register being decremented to zero, changing a pausedstatus to a valid status for a corresponding thread identifier forsubsequent execution of a thread return instruction for completion ofthe created fibers or threads.

For any of the various representative embodiments, the method mayfurther comprise: executing a waiting or nonwaiting fiber joininstruction. For any of the various representative embodiments, themethod may further comprise: executing a fiber join all instruction.

For any of the various representative embodiments, the method mayfurther comprise: executing a non-cached read or load instruction todesignate a general-purpose register for storage of data received from amemory.

For any of the various representative embodiments, the method mayfurther comprise: executing a non-cached write or store instruction todesignate data in a general-purpose register for storage in a memory.

For any of the various representative embodiments, the method mayfurther comprise: assigning a transaction identifier to any load orstore request to memory and to correlate the transaction identifier witha thread identifier.

For any of the various representative embodiments, the method mayfurther comprise: executing a first thread priority instruction toassign a first priority to an execution thread having a correspondingthread identifier. For any of the various representative embodiments,the method may further comprise: executing a second thread priorityinstruction to assign a second priority to an execution thread having acorresponding thread identifier.

For any of the various representative embodiments, the method mayfurther comprise: executing a custom atomic return instruction tocomplete an executing thread of a custom atomic operation.

For any of the various representative embodiments, the method mayfurther comprise: executing a floating point atomic memory operation.

For any of the various representative embodiments, the method mayfurther comprise: executing a custom atomic memory operation.

Numerous other advantages and features of the present invention willbecome readily apparent from the following detailed description of theinvention and the embodiments thereof, from the claims and from theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will bemore readily appreciated upon reference to the following disclosure whenconsidered in conjunction with the accompanying drawings, wherein likereference numerals are used to identify identical components in thevarious views, and wherein reference numerals with alphabetic charactersare utilized to identify additional types, instantiations or variationsof a selected component embodiment in the various views, in which:

FIG. 1 is a block diagram of a representative first embodiment of ahybrid computing system.

FIG. 2 is a block diagram of a representative second embodiment of ahybrid computing system.

FIG. 3 is a block diagram of a representative third embodiment of ahybrid computing system.

FIG. 4 is a block diagram of a representative embodiment of a hybridthreading fabric having configurable computing circuitry coupled to afirst interconnection network.

FIG. 5 is a high-level block diagram of a portion of a representativeembodiment of a hybrid threading fabric circuit cluster.

FIG. 6 is a high-level block diagram of a second interconnection networkwithin a hybrid threading fabric circuit cluster.

FIG. 7 is a detailed block diagram of a representative embodiment of ahybrid threading fabric circuit cluster.

FIG. 8 is a detailed block diagram of a representative embodiment of ahybrid threading fabric configurable computing circuit (tile).

FIGS. 9A and 9B (collectively referred to as FIG. 9 ) are collectively adetailed block diagram of a representative embodiment of a hybridthreading fabric configurable computing circuit (tile).

FIG. 10 is a detailed block diagram of a representative embodiment of amemory control circuit of a hybrid threading fabric configurablecomputing circuit (tile).

FIG. 11 is a detailed block diagram of a representative embodiment of athread control circuit of a hybrid threading fabric configurablecomputing circuit (tile).

FIG. 12 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) forming synchronous domains andrepresentative asynchronous packet network messaging.

FIG. 13 is a block diagram of a representative embodiment of a memoryinterface.

FIG. 14 is a block diagram of a representative embodiment of a dispatchinterface.

FIG. 15 is a block diagram of a representative embodiment of an optionalfirst network interface.

FIG. 16 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) forming synchronous domains andrepresentative asynchronous packet network messaging for performance ofa computation by a hybrid threading fabric circuit cluster.

FIGS. 17A, 17B, and 17C (collectively referred to as FIG. 17 ) is a flowchart of representative asynchronous packet network messaging andexecution by hybrid threading fabric configurable computing circuits(tiles) for performance of the computation of FIG. 16 by a hybridthreading fabric circuit cluster.

FIG. 18 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) forming synchronous domains andrepresentative asynchronous packet network messaging for performance ofa computation by a hybrid threading fabric circuit cluster.

FIGS. 19A, and 19B (collectively referred to as FIG. 19 ) is a flowchart of representative asynchronous packet network messaging andexecution by hybrid threading fabric configurable computing circuits(tiles) for performance of the computation of FIG. 18 by a hybridthreading fabric circuit cluster.

FIG. 20 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) forming synchronous domains andrepresentative asynchronous packet network messaging for performance ofa loop in a computation by a hybrid threading fabric circuit cluster.

FIG. 21 is a flow chart of representative asynchronous packet networkmessaging and execution by hybrid threading fabric configurablecomputing circuits (tiles) for performance of the loop in a computationof FIG. 20 by a hybrid threading fabric circuit cluster.

FIG. 22 is a block diagram of a representative flow control circuit.

FIG. 23 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) forming synchronous domains andrepresentative asynchronous packet network messaging and synchronousmessaging for performance of a loop in a computation by a hybridthreading fabric circuit cluster.

FIG. 24 is a block and circuit diagram of a representative embodiment ofconditional branching circuitry.

FIG. 25 is a high-level block diagram of a representative embodiment ofa hybrid threading processor 300.

FIG. 26 is a detailed block diagram of a representative embodiment of athread memory of the hybrid threading processor.

FIG. 27 is a detailed block diagram of a representative embodiment of anetwork response memory of the hybrid threading processor.

FIG. 28 is a detailed block diagram of a representative embodiment of ahybrid threading processor.

FIGS. 29A and 29B (collectively referred to as FIG. 29 ) are a flowchart of a representative embodiment of a method for self-scheduling andthread control for a hybrid threading processor.

FIG. 30 is a detailed block diagram of a representative embodiment of athread selection control circuitry of the control logic and threadselection circuitry of the hybrid threading processor.

FIG. 31 is a block diagram of a representative embodiment of a portionof the first interconnection network and representative data packets.

FIG. 32 is a detailed block diagram of a representative embodiment ofdata path control circuitry of a hybrid threading processor.

FIG. 33 is a detailed block diagram of a representative embodiment ofsystem call circuitry of a hybrid threading processor and host interfacecircuitry.

FIG. 34 is a block diagram of a representative first embodiment of afirst interconnection network.

FIG. 35 is a block diagram of a representative second embodiment of afirst interconnection network.

FIG. 36 is a block diagram of a representative third embodiment of afirst interconnection network.

FIG. 37 illustrates representative virtual address space formatssupported by the system architecture.

FIG. 38 illustrates a representative translation process for eachvirtual address format.

FIG. 39 illustrates a representative send call example for hybridthreading.

FIG. 40 illustrates a representative send fork example for hybridthreading.

FIG. 41 illustrates a representative send transfer example for hybridthreading.

FIG. 42 illustrates a representative call chain use example for hybridthreading.

DETAILED DESCRIPTION OF REPRESENTATIVE EMBODIMENTS

While the present invention is susceptible of embodiment in manydifferent forms, there are shown in the drawings and will be describedherein in detail specific exemplary embodiments thereof, with theunderstanding that the present disclosure is to be considered as anexemplification of the principles of the invention and is not intendedto limit the invention to the specific embodiments illustrated. In thisrespect, before explaining at least one embodiment consistent with thepresent invention in detail, it is to be understood that the inventionis not limited in its application to the details of construction and tothe arrangements of components set forth above and below, illustrated inthe drawings, or as described in the examples. Methods and apparatusesconsistent with the present invention are capable of other embodimentsand of being practiced and carried out in various ways. Also, it is tobe understood that the phraseology and terminology employed herein, aswell as the abstract included below, are for the purposes of descriptionand should not be regarded as limiting.

I. Hybrid Computing System 100 and Interconnection Networks:

FIGS. 1, 2 and 3 are block diagrams of representative first, second, andthird embodiments of a hybrid computing system 100A, 100B, 100C(collectively referred to as a system 100). FIG. 4 is a block diagram ofa representative embodiment of a hybrid threading fabric (“HTF”) 200having configurable computing circuitry coupled to a firstinterconnection network 150 (also abbreviated and referred to as a“NOC”, as a “Network On a Chip”). FIG. 5 is a high-level block diagramof a portion of a representative embodiment of a hybrid threading fabriccircuit cluster 205 with a second interconnection network 250. FIG. 6 isa high-level block diagram of a second interconnection network within ahybrid threading fabric cluster 205. FIG. 7 is a detailed block diagramof a representative embodiment of a hybrid threading fabric (HTF)cluster 205.

FIG. 8 is a high-level block diagram of a representative embodiment of ahybrid threading fabric configurable computing circuit 210, referred toas a “tile” 210. FIG. 9 is a detailed block diagram of a representativeembodiment of a hybrid threading fabric configurable computing circuit210A, referred to as a “tile” 210A, as a particular representativeinstantiation of a tile 210. Unless specifically referred to as a tile210A, reference to a tile 210 shall mean and refer, individually andcollectively, to a tile 210 and tile 210A. The hybrid threading fabricconfigurable computing circuit 210 is referred to as a “tile” 210because all such hybrid threading fabric configurable computing circuits210, in a representative embodiment, are identical to each other and canbe arrayed and connected in any order, i.e., each hybrid threadingfabric configurable computing circuit 210 can be “tiled” to form ahybrid threading fabric cluster 205.

Referring to FIGS. 1-9 , a hybrid computing system 100 includes a hybridthreading processor (“HTP”) 300, discussed in greater detail below withreference to FIGS. 25-33 , which is coupled through a firstinterconnection network 150 to one or more hybrid threading fabric(“HTF”) circuits 200. It should be understood that term “fabric”, asused herein, means and includes an array of computing circuits, which inthis case are reconfigurable computing circuits. FIGS. 1, 2, and 3 showdifferent system 100A, 100B, and 100C arrangements which includeadditional components forming comparatively larger and smaller systems100, any and all of which are within the scope of the disclosure. Asshown in FIGS. 1 and 2 , which may each be an arrangement suitable for asystem-on-a-chip (“SOC”), for example and without limitation, a hybridcomputing system 100A, 100B, in various combinations as illustrated, mayalso include, optionally, a memory controller 120 which may be coupledto a memory 125 (which also may be a separate integrated circuit), anyof various communication interfaces 130 (such as a PCIe communicationinterface), one or more host processor(s) 110, and a host interface(“HIF”) 115. As shown in FIG. 3 , which may each be an arrangementsuitable for a “chiplet” configuration on a common substrate 101, forexample and without limitation, a hybrid computing system 100C may alsoinclude, optionally, a communication interface 130, with or withoutthese other components. Any and all of these arrangements are within thescope of the disclosure, and collectively are referred to herein as asystem 100. Any of these hybrid computing systems 100 also may beconsidered a “node”, operating under a single operating system (“OS”),and may be coupled to other such local and remote nodes as well.

Each node of a system 100 runs a separate Operating System (OS)instance, controlling the resources of the associated node. Anapplication that spans multiple nodes is executed through thecoordination of the multiple OS instances of the spanned nodes. Theprocess associated with the application running on each node has anaddress space that provides access to node private memory, and to theglobally shared memory that is distributed across nodes. Each OSinstance includes a driver that manages the local node resources. Anapplication's shared address space is managed collectively by the set ofdrivers running on the nodes. The shared address space is allocated aGlobal Space ID (GSID). The number of global spaces that are active atany given time is expected to be relatively small. The GSID is set at 8bits wide.

Hybrid threading, as used herein, refers to the capability to spawnmultiple fibers and threads of computation across different,heterogeneous types of processing circuits (hardware), such as acrossHTF circuits 200 (as a reconfigurable computing fabric) and across aprocessor, such as the HTP 300 or another type of RISC-V processor.Hybrid threading also refers to a programming language/style in which athread of work transitions from one compute element to the next to movethe compute to where the data is located, which is also implemented inrepresentative embodiments. A host processor 110 is typically amulti-core processor, which may be embedded within the hybrid computingsystem 100, or which may be an external host processor coupled into thehybrid computing system 100 via a communication interface 130, such as aPCIe-based interface. These processors, such as the HTP 300 and the oneor more host processor(s) 110, are described in greater detail below.

The memory controller 120 may be implemented as known or becomes knownin the electronic arts. Alternatively, in a representative embodiment,the memory controller 120 may be implemented as described in the relatedapplications. The first memory 125 also may be implemented as known orbecomes known in the electronic arts, and as described in greater detailbelow.

Also in a representative embodiment, the HTP 300 is a RISC-V ISA basedmulti-threaded processor having one or more processor cores 705 havingan extended instruction set, with one or more core control circuits 710and one or more second memories 715, referred to as a core control (orthread control) memories 715, as discussed in greater detail below.Generally, the HTP 300 provides barrel-style, round-robin instantaneousthread switching to maintain a high instruction-per-clock rate.

The HIF 115, for the purposes herein, provides for a host processor 110to send work to the HTP 300 and the HTF circuits 200, and for the HTP300 to send work to the HTF circuits 200, both as “work descriptorpackets” transmitted over the first interconnection network 150. Aunified mechanism is provided to start and end work on an HTP 300 and anHTF circuit 200: “call” work descriptor packets are utilized to startwork on an HTP 300 and an HTF circuit 200, and “return” work descriptorpackets are utilized to end work on an HTP 300 and an HTF circuit 200.The HIF 115 includes a dispatch circuit and queue (abbreviated “dispatchqueue” 105), which also provides management functionality for monitoringthe load provided to and resource availability of the HTF circuits 200and/or HTP 300. When resources are available on the HTF circuits 200and/or HTP 300, the dispatch queue 105 determines the HTF circuit 200and/or HTP 300 resource that is least loaded. In the case of multipleHTF circuit clusters 205 with the same or similar work loading, itchooses an HTF circuit cluster 205 that is currently executing the samekernel if possible (to avoid having to load or reload a kernelconfiguration). Similar functionality of the HIF 115 may also beincluded in an HTP 300, for example, particularly for system 100arrangements which may not include a separate HIF 115. Other HIF 115functions are described in greater detail below. An HIF 115 may beimplemented as known or becomes known in the electronic arts, e.g., asone or more state machines with registers (forming FIFOs, queues, etc.).

The first interconnection network 150 is a packet-based communicationnetwork providing data packet routing between and among the HTF circuits200, the hybrid threading processor 300, and the other optionalcomponents such as the memory controller 120, a communication interface130, and a host processor 110. For purposes of the present disclosure,the first interconnection network 150 forms part of an asynchronousswitching fabric (“AF”), meaning that a data packet may be routed alongany of various paths, such that the arrival of any selected data packetat an addressed destination may occur at any of a plurality of differenttimes, depending upon the routing. This is in contrast with thesynchronous mesh communication network 275 of the second interconnectionnetwork 250 discussed in greater detail below.

FIG. 31 is a diagram of a representative embodiment of a portion of thefirst interconnection network 150 and representative data packets. Inrepresentative embodiment, the first interconnection network 150includes a network bus structure 152 (a plurality of wires or lines), inwhich a first plurality of the network lines 154 are dedicated foraddressing (or routing) data packets (158), and are utilized for settingthe data path through the various crossbar switches, and the remainingsecond plurality of the network lines 156 are dedicated for transmissionof data packets (the data load, illustrated as a train or sequence of“N” data packets 162 ₁ through 162 _(N)) containing operand data,arguments, results, etc.) over the path established through theaddressing lines (first plurality of the network lines 154). Two suchnetwork bus structures 152 are typically provided, into and out of eachcompute resource, as channels, a first channel for receiving data, and asecond channel for transmitting data. A single, first addressing (orrouting) data packet (illustrated as addressing (or routing) data packet158 ₁) may be utilized to establish the routing to a first designateddestination, and may be followed (generally several clock cycles later,to allow for the setting of the switches) by one or more data packets162 which are to be transmitted to the first designated destination, upto a predetermined number of data packets 162 (e.g., up to N datapackets). While that predetermined number of data packets 162 are beingrouted, another, second addressing (or routing) data packet (illustratedas addressing (or routing) data packet 158 ₂) may be transmitted andutilized to establish a routing to a second designated destination, forother, subsequent one or more data packets 162 which will be going tothis second designated destination (illustrated as data packet 162_(N+1)). Accordingly, data communication over the interconnectionnetwork is performed using a split header and payload configuration forpipelining a plurality of communications, the interconnection network150 is adapted to use the split header and payload configuration fordelayed payload switching, and the interconnection network 150 isadapted to use a single header to route a plurality of data payloads asa sequential data burst. In addition, the interconnection network 150 isadapted to interleave a first acknowledgment message to a destination inan unused header field of a second message to the destination, and theinterconnection network 150 is adapted to provide power gating or clockgating based on load requirements.

FIGS. 34-36 are block diagrams of representative first, second, andthird embodiments of a first interconnection network 150, illustratingas examples various topologies of a first interconnection network 150,such as first interconnection networks 150A, 150B, 150C (any and all ofwhich are referred to herein as a first interconnection network 150).The first interconnection network 150 is typically embodied as aplurality of crossbar switches 905, 910 having a folded closconfiguration, illustrated as central (or hub) crossbar switches 905which are coupled through queues 925 to peripheral (or edge) crossbarswitches 910, and with the peripheral crossbar switches 910 coupled inturn (also via queues 925) to a mesh network 920 which provides for aplurality of additional, direct connections 915, such as betweenchiplets, e.g., up, down, left, right, depending upon the system 100embodiment. Numerous network topologies are available and within thescope of this disclosure, such as illustrated in FIGS. 35 and 36 , withthe first interconnection network 150B, 150C further including endpointcrossbar switches 930.

Routing through any of the various first interconnection networks 150includes load balancing, such that packets moving toward the central (orhub) crossbar switches 905 from the peripheral (or edge) crossbarswitches 910 may be routed through any available crossbar switch of thecentral (or hub) crossbar switches 905, and packets moving toward theperipheral (or edge) crossbar switches 910 from the endpoint crossbarswitches 930 may be routed through any available peripheral (or edge)crossbar switches 910, such as routing with a round-robin distributionor a random distribution to any available switch 905, 910. For routingfrom the central (or hub) crossbar switches 905 through the peripheral(or edge) crossbar switches 910 and/or endpoint crossbar switches 930 toany endpoint (or destination), which may be a HTF 200 or HTP 300, amemory controller 120, a host processor 110, etc., the identifier oraddress (e.g., virtual) of the endpoint (or destination) is utilized,typically having an address or identifier with five fields: (a) a first(or horizontal) identifier; (b) a second (or vertical) identifier; (c) athird, edge identifier; (d) a fourth, group identifier; and (e) a fifth,endpoint identifier. The first (or horizontal) identifier and the second(or vertical) identifier are utilized to route to the correctdestination hub, the edge identifier is utilized to route to theselected chip or chiplet edge (of four available edges), the groupidentifier is utilized to route to the selected communication interfacewhich may be at the selected edge, and an endpoint identifier isutilized for any additional routing, such as through endpoint crossbarswitches 930 or the mesh networks 920. For power savings, any of thevarious central (or hub) crossbar switches 905, peripheral (or edge)crossbar switches 910, and endpoint crossbar switches 930 may be powergated or clock gated, to turn off the various switches when routingdemand may be lower and less capacity may be needed and to turn on thevarious switches when routing demand may be higher and greater capacitymay be needed. Additional aspects of the first interconnection network150 are discussed in greater detail below with reference to FIG. 30 .

A first interconnection network 150 packet consists of a fixed genericpacket header, plus a variable sized packet payload. A single packetheader is required per packet and is used to route the packet from asource component within the system 100 to a destination component. Thepayload is variable in size depending on the type of request or responsepacket. Table 1 shows the information contained in the generic headerfor first interconnection network 150 packets, Table 2 shows theinformation contained in a first interconnection network 150 readrequest packet, and Table 3 shows the information contained in a firstinterconnection network 150 read response packet for a 16B read with 8BFlit size.

TABLE 1 First Interconnection Network 150 Generic Packet Header FieldSize Field Name Flit (bits) Field Description DCID 0 10 DestinationComponent ID-used to route the packet from the source component throughthe first interconnection network 150 to the destination component. LEN0 4 Length of packet-Number of data transfers where a data transfer isone clock cycle. VC 0 3 Virtual Channel-Virtual channels are used toseparate communication flows across the fabric such that one flow cannotindefinitely stall another channel causing a deadlock situation. Credit0 1 Ready Credit VC 0 3 ECC 0 6 Error Correcting Code-Provides errorchecking and correcting on the information required to route a packetthrough the NOC. Header 27 Size

TABLE 2 First Interconnection Network 150 Read Request Packet Field SizeField Name Flit (bits) Field Description GPH 0 27 Generic PacketHeader-Common to all first interconnection network 150 packets. ADDR 046 Read Request Address-Set at 48 bits (256 TB) to allow persistentmemory to be mapped into the address space. SCID 0 10 Source ComponentID-used to route response back to requester. REQ 0 8 Request Type-FieldTYPE TID 0 8 Transaction ID-A unique value provided by the source of therequest and returned with the response. ECC 0 8 Error CorrectingCode-Provides error checking and correcting on the information thattraverses end to end through the NOC. This ECC field does not cover theGeneric Packet Header information. Packet Size 107

TABLE 3 First Interconnection Network 150 16B Read Response Packet (with8B Flits) Field Size Field Name Flit (bits) Field Description GPH 0 27Generic Packet Header-Common to all first interconnection network 150packets. DATA 0 64 First 64 bits of data TID 0 8 TID for response packetECC 0 8 Error Correcting Code-Provides error checking and correction forDATA in Flit #0 GPH 1 27 Generic Packet Header-Ignored for Flit != 0DATA 1 64 Second 64 bits of data TID 1 8 TID-Ignored for Flit != 0 ECC 18 Error Correcting Code-Provides error checking and correction for DATAin Flit #1 Packet Size 214

A HTF circuit 200, in turn, typically comprises a plurality of HTFcircuit clusters 205, with each HTF circuit cluster 205 coupled to thefirst interconnection network 150 for data packet communication. EachHTF circuit cluster 205 may operate independently from each of the otherHTF circuit clusters 205. Each HTF circuit cluster 205, in turn,comprises an array of a plurality of HTF reconfigurable computingcircuits 210, which are referred to equivalently herein as “tiles” 210,and a second interconnection network 250. The tiles 210 are embedded inor otherwise coupled to the second interconnection network 250, whichcomprises two different types of networks, discussed in greater detailbelow. In a representative embodiment, each HTF circuit cluster 205 alsocomprises a memory interface 215, an optional first network interface220 (which provides an interface for coupling to the firstinterconnection network 150), and a HTF dispatch interface 225. Thevarious memory interfaces 215, the HTF dispatch interface 225, and theoptional first network interface 220 may be implemented using anyappropriate circuitry, such as one or more state machine circuits, toperform the functionality specified in greater detail below.

The HTP 300 is a barrel style multi-threaded processor that is designedto perform well on applications with high degree of parallelismoperating on sparse data sets (i.e., applications having minimal datareuse). The HTP 300 is based on the open source RISC-V processor, andexecutes in user mode. The HTP 300 includes more RISC-V user modeinstructions, plus a set of custom instructions to allow threadmanagement, sending and receiving events to/from other HTPs 300, HTFcircuits 200 and one or more host processors 110, and instructions forefficient access to memory 125.

Sparse data sets typically cause poor cache hit rates. The HTP 300 withmany threads per HTP processor core 705 allow some threads to be waitingfor response from memory 125 while other threads are continuing toexecute instructions. This style of compute is tolerant of latency tomemory 125 and allows high sustained executed instructions per clock.The event mechanism allows threads from many HTP cores 705 tocommunicate in an efficient manner. Threads pause executing instructionwhile waiting for memory 125 responses or event messages, allowing otherthreads to use the instruction execution resources. The HTP 300 isself-scheduling and event driven, allowing threads to efficiently becreated, destroyed and communicate with other threads. The HTP 300 isdiscussed in greater detail below with reference to FIGS. 25-33 .

II. Hybrid Threading:

The hybrid threading of the system 100 allows compute tasks totransition from a host processor 110, to an HTP 300 and/or HTF 200 onone node, and then on to an HTP 300 or HTF 200 on possibly a differentnode. During this entire sequence of transitioning work from one computeelement to another, all aspects are handled completely in user space.Additionally, the transition of a compute task from an HTP 300 toanother HTP 300 or to an HTF 200 can occur by executing a single HTP 300instruction and without reference to memory 125. This extremelylightweight thread management mechanism allows applications to quicklycreate large numbers of threads to handle parallelizable kernels of anapplication, and then rejoin when the kernel is complete. The HTP 300and HTF 200 compute elements handle compute tasks very differently(RISC-V instruction execution versus data flow), however they bothsupport the hybrid threading approach and can seamlessly interact onbehalf of an application.

Work descriptor packets are utilized to commence work on an HTP 300 anda HTF circuit 200. Receipt of a work descriptor packet by an HTP 300and/or HTF 200 constitutes an “event” which will trigger hardware-basedself-scheduling and subsequent execution of the associated functions orwork, referred to as threads of execution, in the HTP 300 and/or HTF200, without the need for further access to main memory 125. Once athread is started it executes instructions until a thread returninstruction is executed (by the HTP 300) or a return message isgenerated (by the HTF 200). The thread return instruction sends a returnwork descriptor packet to the original caller.

For purposes of the present disclosure, at a high or general level, awork descriptor packet includes: (1) the information needed to route thework descriptor packet to its destination; (2) to initialize a threadcontext for the HTP 300 and/or an HTF circuit 200, such as a programcount (e.g., as a 64-bit address) for where in the stored instructions(stored in instruction cache 740, FIG. 28 or first instruction RAM 315,FIG. 9 , respectively) to commence thread execution; (3) any argumentsor addresses in first memory 125 to obtain arguments or otherinformation which will be used in the thread execution; and (4) a returnaddress for transmission of computation results, for example and withoutlimitation. While referred to in singular form as a single “workdescriptor packet”, as mentioned above, such a work descriptor packet isactually divided into multiple packets for transmission over the firstinterconnection network 150, an addressing (or routing) data packet 158and one or more data packets 162. There can be many different kinds ofwork descriptor packets, depending upon the operations or instructionsto be performed, with many examples illustrated and discussed below. Theinstruction cache 740 and/or first instruction RAM 315 has beenpopulated in advance of any execution, such as in the initial system 100configuration. For an HTF circuit 200, generally, the work descriptorcall packet also will have similar information, such as addressing, apayload (e.g., a configuration, argument values, etc.), a callidentifier (ID), and return information (for the provision of results tothat endpoint, for example), and other information as discussed ingreater detail below.

At a high or general level, and as discussed in much greater detailbelow, a host processor 110 or HTP 300 can initiate a thread on anotherHTP 300 or HTF 200 by sending it a call work descriptor packet. The callinformation includes the destination node, the call's entry instructionaddress, and up to four 64-bit argument values. Each HTP 300 isinitialized to have a pool of stack and context structures. Thesestructures reside in user space. When an HTP 300 receives a call, itselects a stack and context structure from the free pool. The HTP 300then initializes the new thread with the call information and the stackstructure address. At this point, the initialized thread is put into theactive thread queue to begin execution. The steps to initiate a threadon an HTP 300 may be implemented as a hardware state machine (as opposedto executing instructions) to maximize thread creation throughput. Asimilar hardware-based approach exists for initiating work on the HTF200, also as discussed below.

Once a thread is put in the active thread queue on an HTP 300, it willbe selected to execute instructions. Eventually, the thread willcomplete its compute task. At this point, the HTP 300 will send a returnmessage back to the calling processor by executing a single customRISC-V send return instruction. Sending a return is similar to sending acall. The instruction frees the stack and context structure and sends upto four 64-bit parameters back to the calling processor. A calling HTP300 executes a receive return custom RISC-V instruction to receive thereturn. The HTP calling processor 300 copies the return arguments intoISA visible registers for access by the executing thread. The originalsend call includes the necessary information for the called HTP 300 toknow where to send its return. The information consists of the sourceHTP 300 and thread ID of the calling thread.

An HTP 300 has three options for sending a work task to another HTP 300or HTF 200 compute element. These options are to perform a call, fork ortransfer, illustrated in FIGS. 39-41 :

(a) A call (901) initiates a compute task on the remote HTP 300 or HTF200 and pauses further instruction execution until the return (902) isreceived. The return information passed to the remote compute element isused by the remote compute task when it has completed and is ready toreturn.

(b) A fork (903) initiates a compute task on the remote HTP 300 or HTF200 and continues executing instructions. A single thread could initiatemany compute tasks on remote HTP 300 or HTF 200 compute elements usingthe send fork mechanism. The original thread must wait until a return(902) has been received from each forked thread prior to sending itsreturn. The return information passed to the remote compute element isused by the remote compute task when it has completed and is ready toreturn.

(c) A transfer (904) initiates a compute task on a remote HTP 300 or HTF200 and terminates the original thread. The return (902) informationpassed to the remote compute element is the return information from thecall, fork or transfer that initiated the current thread. The send fork(903) includes information to return to the thread that executed thesent fork instruction on a first HTP 300. The send transfer (Xfer)executed on the second HTP 300 includes the information to return to thethread that executed the send fork instruction on the first HTP 300.Essentially, a send transfer just passes on the return information itwas provided when it was initiated. Finally, the thread that executesthe send return on a third or fourth HTP 300 uses the return informationit received to determine the destination for the return.

While the call, fork, and transfer are illustrated in FIGS. 39-41 forcommunication between HTPs 300, an HTP 300 may also send similar workdescriptor packets to an HTF 200, as illustrated in FIG. 42 for a callchain example.

A thread has access to private memory on the local node as well asshared memory on local and remote nodes through references to thevirtual address space. An HTP 300 thread will primarily use the providedinbound call arguments and private memory stack to manipulate datastructures stored in shared memory. Similarly, an HTF 200 thread willuse the inbound call arguments and in-fabric memories to manipulate datastructures stored in shared memory.

An HTP 300 thread is typically provided up to four call arguments and astack when the thread is created. The arguments are located in registers(memory 715, discussed below), and the stack is located in node privatememory. A thread will typically use the stack for thread privatevariables and HTP 300 local calls using the standard stack frame basedcalling approach. An HTP 300 thread also has access to the entirepartitioned global memory of the application. It is expected thatapplication data structures are primarily allocated from the partitionedglobal address space to allow all node compute elements to participatein computations with direct load/store accesses.

Each HTP 300 thread has a context block provided when the thread isinitiated. The context block provides a location in memory 125 to whichthe thread context can be saved when needed. Typically, this will occurfor debugging purposes, and it will occur if more threads are createdthan hardware resources are available to handle them. A user can limitthe number of active threads to prevent a thread from ever writing stateto its memory-based context structure (other than possibly for debuggingvisibility).

An HTF 200 thread is also typically provided up to four call argumentswhen a thread is created. The arguments are placed in in-fabric memorystructures for access by the data flow computations. In-fabric memoriesare also used for thread private variables. An HTF 200 thread has accessto the entire partitioned global memory of the application.

The compute elements of the system 100 have different capabilities thatmake each uniquely suited for specific compute tasks. The host processor110 (either internal or external to device) is designed for lowestpossible latency when executing a single thread. The HTP 300 isoptimized for executing a large set of threads concurrently to providethe highest execution throughput. The HTF 200 is optimized for very highperformance on data flow style kernels. The compute elements have beenarchitected to very efficiently hand off compute tasks from one elementto the next to execute a compute kernel as efficiently as possible. FIG.42 illustrates a representative call chain use example for hybridthreading that leverages each of the compute elements, and shows atraditional hierarchically structured usage model like a simulation.High throughput data intensive applications are likely to use adifferent usage model oriented towards a number of independent streams.

All applications start execution on a host processor 110 (internal orexternal). The host processor 110 will typically make a set of nestedcalls as it decides the appropriate action to take based on inputparameters. Eventually, the application reaches the compute phase of theprogram. The compute phase may best be suited for execution on the hostprocessor 110, or for accelerated execution by calling either the HTP300 and/or HTF 200 compute elements. FIG. 42 shows the host processor110 performing multiple calls (901) to the HTPs 300. Each HTP 300 willtypically fork (903) a number of threads to perform its compute task.The individual threads can perform computation (integer and floatingpoint), access memory (reads, writes), as well as transfer threadexecution to another HTP 300 or HTF 200 (on the same node or a remotenode), such as through calls (901) to an HTF 200. The ability to movethe execution of a kernel to another node can be advantageous byallowing the compute task to be performed near the memory that needs tobe accessed. Performing work on the appropriate node device can greatlyreduce inter-node memory traffic, accelerating the execution of theapplication. It should be noted that an HTF 200 does not make calls tothe host processor 110 or HTPs 300 in representative embodiments, andonly makes calls to HTFs 200 in special situations (i.e., when definedat compile time).

A host processor 110 is able to initiate a thread on an HTP 300 or HTF200 on the local node. For an external host processor 110, the localnode is the node connected to the host via the PCIe or othercommunication interface 130. For an internal host processor 110, thelocal node is the node in which the host processor 110 is embedded. Adescription of how work is initiated by the host processor 110 on an HTPcore 705 is presented. A similar approach is used for initiating work onan HTF 200.

The host processor 110 initiates work on an HTP core 705 by writing awork descriptor to dispatch queue 105 of a host interface (HIF) 115. Thedispatch queue 105 is located in private memory such that the hostprocessor 110 is writing to cached data to optimize host processor 110performance. An entry in the dispatch queue 105 is typically 64 bytes insize, allowing sufficient space for remote call information and up tofour 64-bit parameters. It should be noted that in a representativeembodiment, there is one dispatch queue 105 per application per node.For a 64 node system, there would be 64 operating system instances. EachOS instance would have one or more processes, each with their owndispatch queue 105.

The HIF 115 monitors the write pointer for the dispatch queue 105 todetermine when an entry has been inserted. When a new entry exists, theHIF 115 verifies that space exists in the host processor 110 returnqueue for the 64-byte return message. This check is needed to ensurethat the status for a completed call is not dropped due to lack ofreturn queue space. Assuming return space exists, then the HIF 115 readsthe call entry from the dispatch queue 105 and forwards it on to the HTP300 or HTF 200 as a work descriptor packet. The HTP 300 or HTF 200 thenprocess the work descriptor packet, as discussed in greater detailbelow, and generate a return packet.

The entire process of the host processor 110 starting a new thread on anHTP 300 or HTF 200 requires the call information to be staged throughthe dispatch queue 105 (64 bytes written to the queue, and 64 bytes readfrom the queue), but no other accesses to DRAM memory. Staging the callinformation through the dispatch queue 105 provides a neededbackpressure mechanism. If the dispatch queue 105 becomes full, then thehost processor 110 will pause until progress has been made and adispatch queue 105 entry has become available.

The return packet is transmitted over the first interconnection network150 to the HIF 115. The HIF 115 writes the return packet to an availablereturn queue entry. The host processor 110 will typically beperiodically polling the return queue to complete the call and obtainany returned status. It should be noted that the return queue isaccessed in a FIFO order. If returns must be matched to specific calls,then a runtime library can be used to perform this ordering. For manyapplications, it is sufficient to know that all returns have beenreceived and the next phase of the application can begin.

III. Hybrid Threading Fabric 200:

As an overview, the HTF circuit 200 is a coarse grained reconfigurablecompute fabric comprised of interconnected compute tiles 210. The tiles210 are interconnected with a synchronous fabric referred to as thesynchronous mesh communication network 275, allowing data to traversefrom one tile 210 to another tile 210 without queuing. This synchronousmesh communication network 275 allows many tiles 210 to be pipelinedtogether to produce a continuous data flow through arithmeticoperations, and each such pipeline of tiles 210 connected through thesynchronous mesh communication network 275 for performance of one ormore threads of computation is referred to herein as a “synchronousdomain”, which may have series connections, parallel connections, andpotentially branching connections as well. The first tile 210 of asynchronous domain is referred to herein as a “base” tile 210.

The tiles 210 are also interconnected with an asynchronous fabricreferred to as an asynchronous packet network 265 that allowssynchronous domains of compute to be bridged by asynchronous operations,with all packets on the asynchronous packet network 265 capable of beingcommunicated in a single clock cycle in representative embodiments.These asynchronous operations include initiating synchronous domainoperations, transferring data from one synchronous domain to another,accessing system memory 125 (read and write), and performing branchingand looping constructs. Together, the synchronous and asynchronousfabrics allow the tiles 210 to efficiently execute high level languageconstructs. The asynchronous packet network 265 differs from the firstinterconnection network 150 in many ways, including requiring lessaddressing, being a single channel, being queued with a depth-basedbackpressure, and utilizing packed data operands, such as with a datapath of 128 bits, for example and without limitation. It should be notedthat the internal data paths of the various tiles 210 are also 128 bits,also for example and without limitation. Examples of synchronousdomains, and examples of synchronous domains communicating with eachother over the asynchronous packet network 265, are illustrated in FIGS.16, 18, 20 , for example and without limitation.

In a representative embodiment, under most circumstances, thread (e.g.,kernel) execution and control signaling are separated between these twodifferent networks, with thread execution occurring using thesynchronous mesh communication network 275 to form a plurality ofsynchronous domains of the various tiles 210, and control signalingoccurring using messaging packets transmitted over the asynchronouspacket network 265 between and among the various tiles 210. For example,the plurality of configurable circuits are adapted to perform aplurality of computations using the synchronous mesh communicationnetwork 275 to form a plurality of synchronous domains, and theplurality of configurable circuits are further adapted to generate andtransmit a plurality of control messages over the asynchronous packetnetwork 265, with the plurality of control messages comprising one ormore completion messages and continue messages, for example and withoutlimitation.

In a representative embodiment, the second interconnection network 250typically comprises two different types of networks, each providing datacommunication between and among the tiles 210, a first, asynchronouspacket network 265 overlaid or combined with a second, synchronous meshcommunication network 275, as illustrated in FIGS. 6 and 7 . Theasynchronous packet network 265 is comprised of a plurality of AFswitches 260, which are typically implemented as crossbar switches(which may or may not additionally or optionally have a Clos or FoldedClos configuration, for example and without limitation), and a pluralityof communication lines (or wires) 280, 285, connecting the AF switches260 to the tiles 210, providing data packet communication between andamong the tiles 210 and the other illustrated components discussedbelow. The synchronous mesh communication network 275 provides aplurality of direct (i.e., unswitched, point-to-point) connections overcommunication lines (or wires) 270 between and among the tiles 210,which may be register-staged at the inputs and outputs of the tile 210,but otherwise without queuing between tiles 210, and without requiringformation of a data packet. (In FIG. 6 , to better illustrate theoverlaying of the two networks with tiles 210 embedded in the secondinterconnection network 250, the tiles 210 are represented as thevertices of the synchronous mesh communication network 275, and the AFswitches 260 are illustrated as “Xs”, as indicated.)

Referring to FIG. 8 , a tile 210 comprises one or more configurablecomputation circuits 155, control circuitry 145, one or more memories325, a configuration memory (e.g., RAM) 160, synchronous networkinput(s) 135 (coupled to the synchronous mesh communication network275), synchronous network output(s) 170 (also coupled to the synchronousmesh communication network 275), asynchronous (packet) network input(s)140 (coupled to the asynchronous packet network 265), and asynchronous(packet) network output(s) 165 (also coupled to the asynchronous packetnetwork 265). Each of these various components are shown coupled to eachother, in various combinations as illustrated, over busses 180, 185.Those having skill in the electronic arts will recognize that fewer ormore components may be included in a tile 210, along with any of variouscombinations of couplings, any and all of which are consideredequivalent and within the scope of the disclosure.

Representative examples of each of these various components areillustrated and discussed below with reference to FIG. 9 . For example,in a representative embodiment, the one or more configurable computationcircuits 155 are embodied as a multiply and shift operation circuit (“MSOp”) 305 and an Arithmetic, Logical and Bit Operation circuit (“ALB Op”)310, with associated configuration capabilities, such as throughintermediate multiplexers 365, and associated registers, such asregisters 312, for example and without limitation. Also in arepresentative embodiment, the one or more configurable computationcircuits 155 may include a write mask generator 375 and conditional(branch) logic circuitry 370, also for example and without limitation.Also in a representative embodiment, control circuitry 145 may includememory control circuitry 330, thread control circuitry 335, and controlregisters 340, such as those illustrated for a tile 210A, for exampleand without limitation. Continuing with the examples, synchronousnetwork input(s) 135 may be comprised of input registers 350 and inputmultiplexers 355, synchronous network output(s) 170 may be comprised ofoutput registers 380 and output multiplexers 395, asynchronous (packet)network input(s) 140 may be comprised of AF input queues 360 andasynchronous (packet) network output(s) 165 may be comprised of AFoutput queues 390, and may also include or share an AF message statemachine 345.

Significantly, and as discussed in greater detail below, theconfiguration memory (e.g., RAM) 160 is comprised of configurationcircuitry (such as configuration memory multiplexer 372) and twodifferent configuration stores which perform different configurationfunctions, a first instruction RAM 315 (which is used to configure theinternal data path of a tile 210) and a second instruction andinstruction index memory (RAM) 320, referred to herein as a “spoke” RAM320 (which is used for multiple purposes, including to configureportions of a tile 210 which are independent from a current instruction,to select a current instruction and an instruction of a next tile 210,and to select a master synchronous input, among other things, all asdiscussed in greater detail below).

As illustrated in FIG. 8 and FIG. 9 , the communication lines (or wires)270 are illustrated as communication lines (or wires) 270A and 270B,such that communication lines (or wires) 270A are the “inputs” (inputcommunication lines (or wires)) feeding data into the input registers350, and the communication lines (or wires) 270B are the “outputs”(output communication lines (or wires)) moving data from the outputregisters 380. In a representative embodiment, there are a plurality ofsets or busses of communication lines (or wires) 270 into and out ofeach tile 210, from and to each adjacent tile (e.g., synchronous meshcommunication network 275 up link, down link, left link, and rightlink), and from and to other components for distribution of varioussignals, such as data write masks, stop signals, and instructions orinstruction indices provided from one tile 210 to another tile 210, asdiscussed in greater detail below. Alternatively, and not separatelyillustrated, there may also be various dedicated communication lines,such as for asserting a stop signal, such that a stop signal generatedfrom any tile 210 in a HTF circuit cluster 205 can be received promptly,in a limited number of clock cycles, by all other tiles 210 in the HTFcircuit cluster 205.

It should be noted that there are various fields in the various sets orbusses of communication lines forming the synchronous mesh communicationnetwork 275. For example, FIG. 8 and FIG. 9 illustrate four busses ofincoming and outgoing communication lines (or wires) 270A and 270B,respectively. Each one of these sets of communication lines (or wires)270A and 270B may carry different information, such as data, aninstruction index, control information, and thread information (such asTID, XID, loop dependency information, write mask bits for selection ofvalid bits, etc.). One of the inputs 270A may also be designated as amaster synchronous input, including input internal to a tile 210 (fromfeedback of an output), which can vary for each time slice of a tile210, which may have the data for an instruction index for that tile 210of a synchronous domain, for example and without limitation, discussedin greater detail below.

In addition, as discussed in greater detail below, for any inputreceived on the synchronous mesh communication network 275 and held inone or more input registers 350 (of the synchronous network input(s)135), each tile 210 may transfer that input directly to one or moreoutput registers 380 (of the synchronous network output(s) 170) foroutput (typically on a single clock cycle) to another location of thesynchronous mesh communication network 275, thereby allowing a firsttile 210 to communicate, via one or more intermediate, second tiles 210,with any other third tile 210 within the HTF circuit cluster 205. Thissynchronous mesh communication network 275 enables configuration (andreconfiguration) of a statically scheduled, synchronous pipeline betweenand among the tiles 210, such that once a thread is started along aselected data path between and among the tiles 210, as a synchronousdomain, completion of the data processing will occur within a fixedperiod of time. In addition, the synchronous mesh communication network275 serves to minimize the number of any required accesses to memory125, as accesses to memory 125 may not be required to complete thecomputations for that thread be performed along the selected data pathbetween and among the tiles 210.

In the asynchronous packet network 265, each AF switch 260 is typicallycoupled to a plurality of tiles 210 and to one or more other AF switches260, over communication lines (or wires) 280. In addition, one or moreselected AF switches 260 are also coupled (over communication lines (orwires) 285) to one or more of the memory interface 215, the optionalfirst network interface 220, and the HTF dispatch interface 225. Asillustrated, as an example and without limitation, the HTF circuitcluster 205 includes a single HTF dispatch interface 225, two memoryinterfaces 215, and two optional first network interfaces 220. Also asillustrated, as an example and without limitation, in addition to beingcoupled to the other AF switches 260, one of the AF switches 260 isfurther coupled to a memory interface 215, to an optional first networkinterface 220, and to the HTF dispatch interface 225, while another oneof the AF switches 260 is further coupled to a memory interface 215 andto the optional first network interface 220.

Depending upon the selected embodiment, each of the memory interfaces215 and the HTF dispatch interface 225 may also be directly connected tothe first interconnection network 150, with capability for receiving,generating, and transmitting data packets over both the firstinterconnection network 150 and the asynchronous packet network 265, anda first network interface 220 is not utilized or included in HTF circuitclusters 205. For example, the HTF dispatch interface 225 may beutilized by any of the various tiles 210 for transmission of a datapacket to and from the first interconnection network 150. In otherembodiments, any of the memory interfaces 215 and the HTF dispatchinterface 225 may utilize the first network interface 220 for receiving,generating, and transmitting data packets over the first interconnectionnetwork 150, such as to use the first network interface 220 to provideadditional addressing needed for the first interconnection network 150.

Those having skill in the electronic arts will recognize that theconnections between and among the AF switches 260, the tiles 210, theoptional first network interfaces 220, the memory interfaces 215, andthe HTF dispatch interface 225 may occur in any selected combination,with any selected number of components, and with any all such componentselections and combinations considered equivalent and within the scopeof the disclosure. Those having skill in the electronic arts willrecognize that the division of the HTF circuit 200 into a plurality ofHTF clusters 205 is not required, and merely provides a conceptualdivision for ease of describing the various components and theconnections between and among the various components. For example, whilea HTF circuit cluster 205 is illustrated as having sixteen tiles 210,with four AF switches 260, a single HTF dispatch interface 225, twomemory interfaces 215, and two first network interfaces 220 (optional),more or fewer of any of these components may be included in either orboth a HTF circuit cluster 205 or a HTF circuit 200, and as described ingreater detail below, for any selected embodiment, an HTF circuitcluster 205 may be partitioned to vary the number and type of componentwhich may be active (e.g., powered on and functioning) at any selectedtime.

The synchronous mesh communication network 275 allows multiple tiles 210to be pipelined without the need for data queuing. All tiles 210 thatparticipate in a synchronous domain act as a single pipelined data path.The first tile of such a sequence of tiles 210 forming a singlepipelined data path is referred to herein as a “base” tile 210 of asynchronous domain, and such a base tile 210 initiates a thread of workthrough the pipelined tiles 210. The base tile 210 is responsible forstarting work on a predefined cadence referred to herein as the “spokecount”. As an example, if the spoke count is three, then the base tile210 can initiate work every third clock. It should also be noted thatthe computations within each tile 210 can also be pipelined, so thatparts of different instructions can be performed while otherinstructions are executing, such as data being input for a nextoperation while a current operation is executing.

Each of the tiles 210, the memory interfaces 215, and the HTF dispatchinterface 225 has a distinct or unique address (e.g., as a 5-bit wideendpoint ID), as a destination or end point, within any selected HTFcircuit cluster 205. For example, the tiles 210 may have endpoint IDs of0-15, memory interfaces 215 (0 and 1) may have endpoint IDs of 20 and21, and HTF dispatch interface 225 may have endpoint ID of 18 (with noaddress being provided to the optional first network interface 220,unless it is included in a selected embodiment). The HTF dispatchinterface 225 receives a data packet containing work to be performed byone or more of the tiles 210, referred to as a work descriptor packet,which have been configured for various operations, as discussed ingreater detail below. The work descriptor packet will have one or morearguments, which the HTF dispatch interface 225 will then provide ordistribute to the various tiles, as a packet or message (AF message)transmitted through the AF switches 260, to the selected, addressedtiles 210, and further, will typically include an identification of aregion in tile memory 325 to store the data (argument(s)), and a threadidentifier (“ID”) utilized to track and identify the associatedcomputations and their completion.

Messages are routed from source endpoint to destination endpoint throughthe asynchronous packet network 265. Messages from different sources tothe same destination take different paths and may encounter differentlevels of congestion. Messages may arrive in a different order than whenthey are sent out. The messaging mechanisms are constructed to workproperly with non-deterministic arrival order.

FIG. 13 is a block diagram of a representative embodiment of a memoryinterface 215. Referring to FIG. 13 , each memory interface 215comprises a state machine (and other logic circuitry) 480, one or moreregisters 485, and optionally one or more queues 474. The state machine480 receives, generates, and transmits data packets on the asynchronouspacket network 265 and the first interconnection network 150. Theregisters 485 store addressing information, such as virtual addresses oftiles 210, physical addresses within a given node, and various tables totranslate virtual addresses to physical addresses. The optional queues474 store messages awaiting transmission on the first interconnectionnetwork 150 and/or the asynchronous packet network 265.

The memory interface 215 allows the tiles 210 within a HTF circuitcluster 205 to make requests to the system memory 125, such as DRAMmemory. The memory request types supported by the memory interface 215are loads, stores and atomics. From the memory interface 215perspective, a load sends an address to memory 125 and data is returned.A write sends both an address and data to memory 125 and a completionmessage is returned. An atomic operation sends an address and data tomemory 125, and data is returned. It should be noted that an atomic thatjust receives data from memory (i.e. fetch-and-increment) would behandled as a load request by the memory interface 215. All memoryinterface 215 operations require a single 64-bit virtual address. Thedata size for an operation is variable from a single byte to 64 bytes.Larger data payload sizes are more efficient for the device and can beused; however, the data payload size will be governed by the ability ofthe high level language compiler to detect access to large blocks ofdata.

FIG. 14 is a block diagram of a representative embodiment of a HTFdispatch interface 225. Referring to FIG. 14 , a HTF dispatch interface225 comprises a state machine (and other logic circuitry) 470, one ormore registers 475, and one or more dispatch queues 472. The statemachine 470 receives, generates, and transmits data packets on theasynchronous packet network 265 and the first interconnection network150. The registers 475 store addressing information, such as virtualaddresses of tiles 210, and a wide variety of tables tracking theconfigurations and workloads distributed to the various tiles, discussedin greater detail below. The dispatch queues 472 store messages awaitingtransmission on the first interconnection network 150 and/or theasynchronous packet network 265.

As mentioned above, the HTF dispatch interface 225 receives workdescriptor call packet (messages), such as from the host interface 115,over the first interconnection network 150. The work descriptor callpacket will have various information, such as a payload (e.g., aconfiguration, argument values, etc.), a call identifier (ID), andreturn information (for the provision of results to that endpoint, forexample). The HTF dispatch interface 225 will create various AF datamessages for transmission over the asynchronous packet network 265 tothe tiles 210, including to write data into memories 325, which tile 210will be the base tile 210 (a base tile ID, for transmission of an AFcompletion message), a thread ID (thread identifier or “TID”), and willsend a continuation message to the base tile 210 (e.g., with completionand other counts for each TID), so that the base tile 210 can commenceexecution once it has received sufficient completion messages. The HTFdispatch interface 225 maintains various tables in registers 475 totrack what has been transmitted to which tile 210, per thread ID andXID. As results are generated or executions completed, the HTF dispatchinterface 225 will receive AF data messages (indicating complete andwith data) or AF completion messages (indicating completion but withoutdata). The HTF dispatch interface 225 has also maintained various counts(in registers 475) of the number of completion and data messages it willneed to receive to know that kernel execution has completed, and willthen assemble and transmit the work descriptor return data packets, withthe resulting data, a call ID, the return information (e.g., address ofthe requestor), via the first interconnection network 150, and frees theTID. Additional features and functionality of the HTF dispatch interface225 are described in greater detail below.

It should be noted, as mentioned above, that multiple levels (ormultiple types) of TIDs may be and typically are utilized. For example,the HTF dispatch interface 225 allocates a first type of TID, from apool of TIDs, which it transmits to a base tile 210. The base tile 210,in turn, may allocate additional TIDs, such as second and third types ofTIDs, such as for tracking the threads utilized in loops and nestedloops, for example and without limitation. These different TIDs then canalso be utilized to access variables which are private to a given loop.For example, a first type of TID may be used for an outer loop, andsecond and third types of TIDs may be utilized to track iterations ofnested loops.

It should also be noted that a separate transaction ID is utilized fortracking various memory requests over the first interconnection network150.

FIG. 15 is a block diagram of a representative embodiment of an optionalfirst network interface. Referring to FIG. 15 , when included each firstnetwork interface 220 comprises a state machine (and other logiccircuitry) 490 and one or more registers 495. The state machine 490receives, generates, and transmits data packets on the asynchronouspacket network 265 and the first interconnection network 150. Theregisters 495 store addressing information, such as virtual addresses oftiles 210, physical addresses within a given node, and various tables totranslate virtual addresses to physical addresses.

Referring again to FIG. 9 , a representative HTF reconfigurablecomputing circuit (tile) 210A comprises at least one multiply and shiftoperation circuit (“MS Op”) 305, at least one Arithmetic, Logical andBit Operation circuit (“ALB Op”) 310, a first instruction RAM 315, asecond, instruction (and index) RAM 320 referred to herein as a “spoke”RAM 320, one or more tile memory circuits (or memory) 325 (illustratedas memory “0” 325A, memory “1” 325B, through memory “N” 325C, andindividually and collectively referred to as memory 325 or tile memory325). In addition, as previously mentioned, a representative tile 210Aalso typically includes input registers 350 and output registers 380coupled over communication lines (or wires) 270A, 270B to thesynchronous mesh communication network 275, and AF input queues 360 andAF output queues 390 coupled over the communication lines (or wires) 280of the asynchronous packet network 265 to the AF switches 260. Controlcircuits 145 are also typically included in a tile 210, such as memorycontrol circuitry 330, thread control circuitry 335, and controlregisters 340 illustrated for a tile 210A. For decoding and forpreparing (assembling) data packets received from or provided to theasynchronous packet network 265, an AF message state machine 345 is alsotypically included in a tile 210. As part of configurability of the tile210, one or more multiplexers are typically included, illustrated asinput multiplexer 355, output multiplexer 395, and one or moreintermediate multiplexer(s) 365 for selection of the inputs to the MS Op305 and the ALB Op 310. Optionally, other components may also beincluded in a tile 210, such as conditional (branch) logic circuit 370,write mask generator 375, and flow control circuit 385 (which isillustrated as included as part of the AF output queues 390, and whichmay be provided as a separate flow control circuit, equivalently). Thecapabilities of the MS Op 305 and the ALB Op 310 are described ingreater detail below.

The synchronous mesh communication network 275 transfers informationrequired for the synchronous domain to function. The synchronous meshcommunication network 275 includes the fields specified below. Inaddition, many of the parameters used in these fields are also stored inthe control registers 340, and are assigned to a thread to be executedin the synchronous domain formed by a plurality of tiles 210. Thespecified fields of the synchronous mesh communication network 275include:

1. Data, typically having a field width of 64 bits, and comprisingcomputed data being transferred from one tile 210 to the next tile 210in a synchronous domain.

2. An instruction RAM 315 address, abbreviated “INSTR”, typically havinga field width of 8 bits, and comprising an instruction RAM 315 addressfor the next tile 210. The base tile 210 specifies the instruction RAM315 address for the first tile 210 in the domain. Subsequent tiles 210can pass the instruction unmodified, or can conditionally change theinstruction for the next tile 210 allowing conditional execution (i.e.if-then-else or switch statements), described in greater detail below.

3. A thread identifier, referred to herein as a “TID”, typically havinga field width of 8 bits, and comprising a unique identifier for threadsof a kernel, with a predetermined number of TIDs (a “pool of TIDs”)stored in the control registers 340 and potentially available for use bya thread (if not already in use by another thread). The TID is allocatedat a base tile 210 of a synchronous domain and can be used as a readindex into the tile memory 325. The TID can be passed from onesynchronous domain to another through the asynchronous packet network265. As there are a finite number of TIDs available for use, to performother functions or computations, eventually the TID should be freed backto the allocating base tile's TID pool for subsequent reuse. The freeingis accomplished using an asynchronous fabric message transmitted overthe asynchronous packet network 265.

4. A transfer identifier referred to as an “XID”, typically having afield width of 8 bits, and comprising a unique identifier fortransferring data from one synchronous domain to another, with apredetermined number of XIDs (a “pool of XIDs”) stored in the controlregisters 340 and potentially available for use by a thread (if notalready in use by another thread). The transfer may be a direct write ofdata from one domain to another, as an “XID_WR”, or it may be the resultof a memory 125 read (as an “XID_RD”) where the source domain sends avirtual address to memory 125 and the destination domain receives memoryread data. The XID_WR is allocated at the base tile 210 of the sourcedomain. The XID_WR in the source domain becomes the XID_RD in thedestination domain. The XID_WR can be used as a write index for tilememory 325 in the destination domain. XID_RD is used in the destinationdomain as a tile memory 325 read index. As there are a finite number ofXIDs available for use, to perform other functions or computations,eventually the XID should be freed back to the allocating base tile'sXID pool for subsequent reuse. The destination domain should free theXID by sending an asynchronous message to the source domain's base tile210, also over the asynchronous packet network 265.

The synchronous mesh communication network 275 provides both data andcontrol information. The control information (INSTR, XID, TID) is usedto setup the data path, and the DATA field can be selected as a sourcefor the configured data path. Note that the control fields are requiredmuch earlier (to configure the data path) than the data field. In orderto minimize the synchronous domain pipeline delay through a tile 210,the control information arrives at a tile 210 a few clock cycles earlierthan the data.

A particularly inventive feature of the architecture of the HTF circuit200 and its composite HTF circuit clusters 205 and their composite tiles210 is the use of two different configuration RAMs, the instruction RAM315 for data path configuration, and the spoke RAM 320 for multipleother functions, including configuration of portions of a tile 210 whichare independent from any selected or given data path, selection of datapath instructions from the instruction RAM 315, selection of the mastersynchronous input (among the available inputs 270A) for each clockcycle, and so on. As discussed in greater detail below, this novel useof both an instruction RAM 315 and an independent spoke RAM 320 enables,among other things, dynamic self-configuration and self-reconfigurationof the HTF circuit cluster 205 and of the HTF circuit 200 as a whole.

Each tile has an instruction RAM 315 that contains configurationinformation to setup the tile 210 data path for a specific operation,i.e., data path instructions that determine, for example, whether amultiplication, a shift, an addition, etc. will be performed in a giventime slice of the tile 210, and using which data (e.g., data from amemory 325, or data from an input register 350). The instruction RAM 315has multiple entries to allow a tile 210 to be time sliced, performingmultiple, different operations in a pipelined synchronous domain, withrepresentative pipeline sections 304, 306, 307, 308, and 309 of a tile210 illustrated in FIG. 9 . Any given instruction may also designatewhich inputs 270A will have the data and/or control information to beutilized by that instruction. Additionally, each time slice couldconditionally perform different instructions depending on previous tile210 time slice data dependent conditional operations, discussed withreference to FIG. 24 . The number of entries within the instruction RAM315 typically will be on the order of 256. The number may changedepending on the experience gained from porting kernels to the HTF 200.

The supported instruction set should match the needs of the targetapplications, such as for applications having data types of 32 and64-bit integer and floating point values. Additional applications suchas machine learning, image analysis, and 5G wireless processing may beperformed using the HTF 200. This total set of applications would need16, 32 and 64-bit floating point, and 8, 16, 32 and 64-bit integer datatypes. The supported instruction set needs to support these data typesfor load, store and arithmetic operations. The operations supported needto allow a compiler to efficiently map high level language source totile 210 instructions. In a representative embodiment, the tiles 210support the same instruction set as a standard high performanceprocessor, including single instruction multiple data (SIMD) instructionvariants.

The spoke RAM 320 has multiple functions, and in representativeembodiments, one of those functions is to be utilized to configure partsof (a time slice of) the tile 210 that is or are independent of thecurrent instruction for the data path, i.e., the tile 210 configurationsheld in the spoke RAM 320 can be used to configure invariant parts ofthe configuration of the tile 210, e.g., those settings for the tile 210which remain the same across different data path instructions. Forexample, the spoke RAM 320 is used to specify which input (e.g., one ofseveral sets of input communication lines 270A or input registers 350)of the tile 210 is the master synchronous input for each clock cycle, asthe selection control of input multiplexer(s) 355. This is significantbecause the instruction index to select an instruction (from theinstruction RAM 315) for a given time slice of the tile 210, and athread ID (TID), is provided on the master synchronous input. As aresult, even if the actual instruction index provided on an input 270Ato a given tile 210 may be varied (as described with reference to FIG.24 ), what set of inputs 270A that will have that selected instructionindex is not varied, so that any given tile 210 knows in advance whatinput it will use to receive a selected instruction index, independentlyof the instruction specified by that selected instruction index. Theconfigurations held in the spoke RAM 320 also usually designate whichoutputs 270B will be utilized for the selected instruction (or timeslice). The spoke RAM 320 read address input, i.e., the spoke index,comes from a counter that counts (modulo) from zero to the spoke countminus one. All tiles 210 within an HTF circuit cluster 205 generallyshould have the same spoke RAM input value each clock to have propersynchronous domain operation. The spoke RAM 320 also stores instructionindices and is also utilized to select instructions from the instructionRAM 315, so that a series of instructions may be selected for executionby the tile 210 as the count of the spoke RAM 320 changes, for a basetile 210 of a synchronous domain. For subsequent tiles in thesynchronous domain, the instruction index may be provided by a previoustile 210 of the synchronous domain. This aspect of the spoke RAM 320 isalso discussed with reference to FIG. 24 , as the spoke RAM 320 ishighly inventive, enabling dynamic self-configuration andreconfiguration of a HTF circuit cluster 205.

The spoke RAM 320 also specifies when a synchronous input 270A is to bewritten to tile memory 325. This situation occurs if multiple inputs arerequired for a tile instruction, and one of the inputs arrives early.The early arriving input can be written to tile memory 325 and thenlater read from the memory 325 when the other inputs have arrived. Thetile memory 325, for this situation, is accessed as a FIFO. The FIFOread and write pointers are stored in the tile memory region ram.

Each tile 210 contains one or more memories 325, and typically each arethe width of the data path (64-bits), and the depth will be in the rangeof 512 to 1024 elements, for example. The tile memories 325 are used tostore data required to support data path operations. The stored data canbe constants loaded as part of a kernel's cluster 205 configuration, orvariables calculated as part of the data flow. The tile memory 325 canbe written from the synchronous mesh communication network 275 as eithera data transfer from another synchronous domain, or the result of a loadoperation initiated by another synchronous domain. The tile memory isonly read via synchronous data path instruction execution.

Tile memory 325 is typically partitioned into regions. A small tilememory region RAM stores information required for memory region access.Each region represents a different variable in a kernel. A region canstore a shared variable (i.e., a variable shared by all executingthreads). A scalar shared variable has an index value of zero. An arrayof shared variables has a variable index value. A region can store athread private variable indexed by the TID identifier. A variable can beused to transfer data from one synchronous domain to the next. For thiscase, the variable is written using the XID_WR identifier in the sourcesynchronous domain, and read using the XID_RD identifier in thedestination domain. Finally, a region can be used to temporarily storedata produced by a tile 210 earlier in the synchronous data path untilother tile data inputs are ready. For this case, the read and writeindices are FIFO pointers. The FIFO pointers are stored in the tilememory region RAM.

The tile memory region RAM typically contains the following fields:

1. A Region Index Upper, which are the upper bits of a tile memoryregion index. The lower index bits are obtained from an asynchronousfabric message, the TID, XID_WR or XID_RD identifiers, or from the FIFOread/write index values. The Region Index Upper bits are OR′ed with thelower index bits to produce the tile memory 325 index.

2. A Region SizeW, which is the width of a memory region's lower index.The memory region's size is 2^(SizeW) elements.

3. A Region FIFO Read Index, which is the read index for a memory regionacting as a FIFO.

4. A Region FIFO Write Index, which is the write index for a memoryregion acting as a FIFO. The tile performs compute operations for theHTF 200.

The compute operations are performed by configuring the data path withinthe tile 210. There are two functional blocks that perform all computefor the tile 210: the multiply and shift operation circuit (“MS Op”)305, and the Arithmetic, Logical and Bit Operation circuit (“ALB Op”)310. The MS Op 305 and ALB Op 310 are under the control of theinstructions from the instruction RAM 315, and can be configured toperform two pipelined operations such as a Multiply and Add, or Shiftand AND, for example and without limitation. (In a representativeembodiment, all devices that support the HTF 200 would have the completesupported instruction set. This would provide binary compatibilityacross all devices. However, it may be necessary to have a base set offunctionality and optional instruction set classes to meet die sizetradeoffs. This approach is similar to how the RISC-V instruction sethas a base and multiple optional instruction subsets.) As illustrated inFIG. 9 , the outputs of the MS Op 305 and ALB Op 310 may be provided toregisters 312, or directly to other components, such as outputmultiplexers 395, conditional logic circuitry 370, and/or write maskgenerator 375.

The various operations performed by the MS Op 305 include, for exampleand without limitation: integer and floating point multiply, shift, passeither input, signed and unsigned integer multiply, signed and unsignedshift right, signed and unsigned shift left, bit order reversal,permutations, any and all of these operations as floating pointoperations, and interconversions between integer and floating point,such as double precision floor operations or convert floating point tointeger. The various operations performed by the ALB Op 310 include, forexample and without limitation: signed and unsigned addition, absolutevalue, negate, logical NOT, add and negate, subtraction A-B, reversesubtraction B-A, signed and unsigned greater than, signed and unsignedgreater than or equal to, signed and unsigned less than, signed andunsigned less than or equal to, comparison (equal or not equal to),logical operations (AND, OR, XOR, NAND, NOR, NOT XOR, AND NOT, OR NOT,any and all of these operations as floating point operations, andinterconversions between integer and floating point, such as flooroperations or convert floating point to integer.

The inputs to the ALB Op 310 and the MS Op 305 are from either thesynchronous tile inputs 270A (held in registers 350), from the internaltile memories 325, or from a small constant value provided within theinstruction RAM 315. The following Table 4, showing tile 210 data pathinput sources, lists the typical inputs for the ALB Op 310 and the MS Op305.

TABLE 4 Source Name Source Description SYNC_U Synchronous meshcommunication network 275 up link SYNC_D Synchronous mesh communicationnetwork 275 down link SYNC_L Synchronous mesh communication network 275left link SYNC_R Synchronous mesh communication network 275 right linkTILE_OUT Output of ALB Op 310 within the tile 210. RDMEM0_T Memory 0read data. Memory 325 region is indexed using TID from the MasterSynchronous Interface. RDMEM0_X Memory 0 read data. Memory 325 region isindexed using XID from the Master Synchronous Interface. RDMEM0_C Memory0 read data. Memory 325 region is indexed using instruction ram constantvalue. RDMEM0_V Memory 0 read data. Memory 325 region is indexed usingvalue received from a synchronous input, as variable indexing. RDMEM0_FMemory 0 read data. Memory 325 region is read using FIFO ordering.RDMEM0_Z Memory 0 read data. Memory 325 region is indexed using thevalue zero. RDMEM1_T Memory 1 read data. Memory 325 region is indexedusing TID from the Master Synchronous Interface. RDMEM1_X Memory 1 readdata. Memory 325 region is indexed using XID from the Master SynchronousInterface. RDMEM1_C Memory 1 read data. Memory 325 region is indexedusing instruction ram constant value. RDMEM1_V Memory 1 read data.Memory 325 region is indexed using value received from a synchronousinput, as variable indexing. RDMEM1_F Memory 1 read data. Memory 325region is read using FIFO ordering. RDMEM1_Z Memory 1 read data. Memory325 region is indexed using the value zero. CONST The data path input isthe zero extended constant value within an instruction. ITER_IDX Thedata path input is the zero extended loop iteration value, described ingreater detail below. ITER_W The data path input is the zero extendedloop iterator width value. See the loop section for more information.

Each of the outputs 270B of a tile 210, as part of the communicationlines 270 of the synchronous mesh communication network 275, areindividually enabled allowing clock gating of the disabled outputs. Theoutput of the ALB Op 310 can be sent to multiple destinations, shown inTable 5.

TABLE 5 Destination Name Destination Description SYNC_U Synchronous meshcommunication network 275 up link SYNC_D Synchronous mesh communicationnetwork 275 down link SYNC_L Synchronous mesh communication network 275left link SYNC_R Synchronous mesh communication network 275 right linkWRMEM0_Z Write Memory 0. Memory 325 region is written using the valuezero as the index. WRMEM0_C Write Memory 0. Memory 325 region is writtenusing the instruction constant field as the index. WRMEM0_T Write Memory0. Memory 325 region is written using the TID value as the index.

At a high level, and all as representative examples without limitation,the general operation of a tile 210 is as follows. The synchronous meshcommunication network 275 and synchronous domains of the various tiles210 are all scheduled as part of the program compilation and when theconfiguration is loaded into the system. Unless paused or stopped, atile 210 can execute its operations when all required inputs are ready,for example, data variables are in input registers 350 or memory 325,and are available to be read or taken from the registers or memory andused, as described in greater detail below. In a representativeembodiment, each pipeline stage may operate in a single clock cycle,while in other representative embodiments, additional clock cycles maybe utilized per pipeline stage. In a first pipeline stage 304, data isinput, such as into the AF input queues 360 and input registers 350, andoptionally directly into the memory 325. In a next pipeline stage 306,AF messages are decoded by AF state machine 345 and moved into memory325; the AF state machine 345 reads data from memory 325 or receivedfrom the output multiplexers 395 and generates a data packet fortransmission over the asynchronous packet network 265; data in the inputregisters 350 is moved into memory 325 or selected as operand data(using input multiplexers 355 and intermediate multiplexers 365), orpassed directly to output registers 380 for output on the synchronousmesh communication network 275; for example. In one or more of the nextpipeline stages 307 and 308, computations are performed by the ALB Op310 and/or the MS Op 305, write masks may be generated by write maskgenerator 375, and instructions (or instruction indices) may be selectedbased on test conditions in conditional (branch) logic circuitry 370. Ina next pipeline stage 309, outputs are selected using outputmultiplexers 395, and output messages (which may have been stored in theAF output queues 390) are transmitted on the asynchronous packet network265, and output data in any of the output registers 380 are transmittedon the synchronous mesh communication network 275.

FIG. 10 is a detailed block diagram of a representative embodiment of amemory control circuit 330 (with associated control registers 340) of ahybrid threading fabric configurable computing circuit (tile) 210. FIG.10 shows a diagram of the tile memory 325 read indexing logic of thememory control circuit 330, and is duplicated for each memory 325 (notseparately illustrated). The instruction RAM 315 has a field thatspecifies which region of the tile memory 325 is being accessed, and afield that specifies the access indexing mode. The memory region RAM 405(part of the control registers 340) specifies a region read mask thatprovides the upper memory address bits for the specific region. The maskis OR′ed (OR gate 408) in with the lower address bits supplied by theread index selection mux 403. The memory region RAM 405 also containsthe read index value when the tile memory 325 is accessed in FIFO mode.The read index value in the RAM 405 is incremented and written back whenaccessing in FIFO mode. The memory region RAM 405, in variousembodiments, may also maintain a top of TID stack through nested loops,described below.

FIG. 10 also shows the control information (INSTR, XID, TID) for thesynchronous mesh communication network 275 is required a few clocksearlier than the data input. For this reason, the control information issent out of the previous tile 210 a few clocks prior to sending thedata. This staging of synchronous mesh communication network 275information reduces the overall pipeline stages per tile 210, but itmakes it challenging to use a calculated value as an index to the tilememories 325. Specifically, the synchronous mesh communication network275 data may arrive too late to be used as an index into the tilememories 325. The architected solution to this problem is to provide thecalculated index from a previous tile 210 in a variable index registerof the control registers 340. Later, another input 270A causes thevariable index register to be used as a tile memory 325 index.

The asynchronous packet network 265 is used to perform operations thatoccur asynchronous to a synchronous domain. Each tile 210 contains aninterface to the asynchronous packet network 265 as shown in FIG. 9 .The inbound interface (from communication lines 280A) is the AF inputqueues 360 (as a FIFO) to provide storage for messages that cannot beimmediately processed. Similarly, the outbound interface (tocommunication lines 280B) is the AF output queues 390 (as a FIFO) toprovide storage for messages that cannot be immediately sent out. Themessages over the asynchronous packet network 265 can be classified aseither data messages or control. Data messages contain a 64-bit datavalue that is written to one of the tile memories 325. Control messagesare for controlling thread creation, freeing resources (TID or XID), orissuing external memory references. The following Table 6 lists theasynchronous packet network 265 outbound message operations:

TABLE 6 Operation Name Operation Description FREE_XID A message sent tothe base tile 210 of a synchronous domain to free an XID RD. FREE_TID Amessage sent to the base tile 210 of a synchronous domain to free a TID.CONT_X A first type of continuation message sent to the base tile 210 ofa synchronous domain to indicate that a thread should be initiated aftera specified number of completion messages have been received. CONT_T Asecond type of continuation message sent to the base tile 210 of asynchronous domain to indicate that a thread should be initiated after aspecified number of completion messages have been received. INNER_LOOP Amessage sent to initiate an inner loop of a strip mined loop construct.The message specifies the number of loop iterations to perform. A workthread is initiated for each iteration. The iteration index is availablewithin the base tile 210 as an input to the data path source multiplexer365 (ITER_IDX). OUTER_LOOP A message sent to initiate an outer loop of astrip mined loop construct. The message specifies the number of loopiterations to perform. A work thread is initiated for each iteration.The iteration index is available within the base tile 210 as an input tothe data path source multiplexer 365 (ITER_IDX). COMP A completionmessage is sent to indicate a synchronous domain work thread hascompleted. A base tile 210 counts the received completion messages inconjunction with receiving a call or continue message in order to allowa subsequent work thread to be initiated. The message sends the TIDidentifier as the pause table index, described below. CALL A callmessage is sent to continue a work thread on the same or anothersynchronous domain. A TID and/or an XID can optionally be allocated whenthe work thread is initiated. CALL_DATA A call data message is sent tocontinue a work thread on the same or another synchronous domain. A TIDand/or an XID can optionally be allocated when the work thread isinitiated. This message sends 128 bits (two 64-bit values) to be writtento tile memory 325 within the base tile 210, along with a maskindicating which bytes of the 128 bit value to write. This is generallyalso the case for all asynchronous messages DATA_R A message is sent towrite to tile memory 325 of the destination tile 210. The TID value isused to specify the write index for the destination tile’s memory. Acompletion message is sent once the tile memory 325 is written tospecified base tile 210. DATA_X A message is sent to write to tilememory 325 of the destination tile 210. The XID WR value is used tospecify the write index for the destination tile’s memory 325. Acompletion message is sent once the tile memory 325 is written tospecified base tile 210. LD_ADDR_T A message is sent to the MemoryInterface 215 to specify the address for a memory load operation. TheTID identifier is used as the write index for the destination tile’smemory. LD_ADDR_X A message is sent to the Memory Interface 215 tospecify the address for a memory load operation. The XID WR identifieris used as the write index for the destination tile’s memory. LD_ADDR_ZA message is sent to the Memory Interface 215 to specify the address fora memory load operation. Zero is used as the write index for thedestination tile’s memory. ST_ADDR A message is sent to the MemoryInterface 215 to specify the address for a memory store operation.ST_DATA A message is sent to the Memory Interface 215 to specify thedata for a memory store operation.

The asynchronous packet network 265 allows messages to be sent andreceived from tiles 210 in different synchronous domains. There are fewsituations where it makes sense for a synchronous domain to send amessage to itself, such as when a synchronous domain's base tile 210allocates a TID, and the TID is to be freed by that same synchronousdomain.

The asynchronous packet network 265 can become congested if thesynchronous domains of the various tiles 210 generate and send messagesfaster than they can be received, routed and processed by theasynchronous packet network 265 or receiving endpoints. In thissituation, a backpressure mechanism is provided to alleviate any suchcongestion. FIG. 22 is a block diagram of a representative flow controlcircuit 385. Generally, there is at least one flow control circuit 385per HTF circuit cluster 205. The tile 210 asynchronous fabric outputqueues 390 will hold messages as they wait to be sent on theasynchronous packet network 265. A predetermined threshold is providedfor the output queue 390 that, when reached, will cause an output queue390 of a tile 210 to generate an indicator, such as setting a bit, whichis asserted as a “stop” signal 382 on a communication line 384 providedto the flow control circuit 385. Each communication line 384 from a tile210 in a HTF circuit cluster 205 is provided to the flow control circuit385. The flow control circuit 385 has one or more OR gates 386, whichwill continue to assert the stop signal 382 on communication line 388distributed to all tiles 210 within the affected HTF circuit cluster205, for as long as any one of the tiles 210 is generating a stop signal382.

The stop signal 382 may be distributed over a dedicated communicationline 388 which is not part of either the synchronous mesh communicationnetwork 275 or the asynchronous packet network 265 as illustrated, orover the synchronous mesh communication network 275. In a representativeembodiment, there is a single stop signal 382 that all tile outputqueues 390 within a HTF circuit cluster 205 can assert, and all tiles210 within that HTF circuit cluster 205 are held (paused or stopped)when a stop signal 382 is asserted. This stop signal 382 continues toallow all AF input queues 360 to receive AF messages and packets,avoiding deadlock, but also causes all synchronous domain pipelines tobe held or paused (which also prevents the generation of additional AFdata packets). The stop signal 382 allows the asynchronous packetnetwork 265 to drain the tile 210 output queues 390 to the point wherethe number of messages in the output queue 390 (of the triggering outputqueue(s) 390) has fallen below the threshold level. Once the size of theoutput queue 390 has fallen below the threshold level, then the signalover the communication line 384 is returned to zero (the stop signal 382is no longer generated) for that tile 210. When that has happened forall of the tiles 210 in the HTF circuit cluster 205, the signal oncommunication line 388 also returns to zero, meaning the stop signal isno longer asserted, and ending the stop or pause on the tiles 210.

The first or “base” tile 210 of a synchronous domain has theresponsibility to initiate threads of work through the multi-tile 210synchronous pipeline. A new thread can be initiated on a predeterminedcadence. The cadence interval referred to herein as the “spoke count”,as mentioned above. For example, if the spoke count is three, then a newthread of work can be initiated on the base tile 210 every three clocks.If starting a new thread is skipped (e.g., no thread is ready to start),then the full spoke count must be waited before another thread can bestarted. A spoke count greater than one allows each physical tile 210 tobe used multiple times within the synchronous pipeline. As an example,if a synchronous domain is executed on a single physical tile and thespoke count is one, then the synchronous domain can contain only asingle tile time slice. If, for this example the spoke count is four,then the synchronous domain can contain four tile time slices.Typically, a synchronous domain is executed by multiple tiles 210interconnected by the synchronous links of the synchronous meshcommunication network 275. A synchronous domain is not restricted to asubset of tiles 210 within a cluster 205, i.e., multiple synchronousdomains can share the tiles 210 of a cluster 205. A single tile 210 canparticipate in multiple synchronous domains, e.g., spoke 0, a tile 210works on synchronous domain “A”; spoke 1, that tile 210 works onsynchronous domain “B”; spoke 2, that tile 210 works on synchronousdomain “A”; and spoke 3, that tile 210 works on synchronous domain “C”.Thread control for a tile is described below with reference to FIG. 11 .

FIG. 11 is a detailed block diagram of a representative embodiment of athread control circuit 335 (with associated control registers 340) of ahybrid threading fabric configurable computing circuit (tile) 210.Referring to FIG. 11 , several registers are included within the controlregisters 340, namely, a TID pool register 410, an XID pool register415, a pause table 420, and a completion table 422. In variousembodiments, the data of the completion table 422 may be equivalentlyheld in the pause table 420, and vice-versa. The thread controlcircuitry 335 includes a continue queue 430, a reenter queue 445, athread control multiplexer 435, a run queue 440, an iteration increment447, an iteration index 460, and a loop iteration count 465.Alternatively, the continue queue 430 and the run queue 440 may beequivalently embodied in the control registers 340.

FIG. 12 is a diagram of tiles 210 forming first and second synchronousdomains 526, 538 and representative asynchronous packet networkmessaging. One difficulty with having an asynchronous packet network 265is that required data may arrive at tiles 210 at different times, whichcan make it difficult to ensure that a started thread can run tocompletion with a fixed pipeline delay. In representative embodiments,the tiles 210 forming a synchronous domain do not execute a computethread until all resources are ready, such as by having the requireddata available, any required variables, etc., all of which have beendistributed to the tiles over the asynchronous packet network 265, andtherefore may have arrived at the designated tile 210 at any of varioustimes. In addition, data may have to be read from system memory 125 andtransferred over the asynchronous packet network 265, and therefore alsomay have arrived at the designated tile 210 at any of various times.

To provide for a thread to run to completion with a fixed pipelinedelay, the representative embodiments provide a completion table 422 (orpause table 420) indexed by a thread's TID at the base tile 210 of asynchronous domain. The completion table 422 (or pause table 420)maintains a count of dependency completions that must be received priorto initiating execution of the thread. The completion table 422 (orpause table 420) includes a field named the “completion count”, which isinitialized to zero at reset. Two types of AF messages are used tomodify the count field. The first message type is a thread start orcontinue message, and increments the field by a count indicating thenumber of dependences that must be observed before a thread can bestarted in the synchronous domain. The second AF message type is acompletion message and decrements the count field by one indicating thata completion message was received. Once a thread start message isreceived, and the completion count field reaches zero, then the threadis ready to be started.

As illustrated in FIG. 12 , a tile 210B of a first synchronous domain526 has transmitted an AF memory load message (293) to the memoryinterface 215 over the asynchronous packet network 265, which in turnwill generate another message (296) to system memory 125 over the firstinterconnect network 150 to obtain the requested data (returned inmessage 297). That data, however, is to be utilized by and istransmitted (message 294) to a tile 201E in the second synchronousdomain 538. As the first synchronous domain 526 has completed itsportion of the pipeline, one of the tiles (210C) in the firstsynchronous domain 526 transmits an AF continue message (291) to thebase tile 210D of the second synchronous domain 538. That AF continuemessage (291) includes the TID of the thread of the first synchronousdomain 526 (e.g., TID=1, which was also included in the other messagesof FIG. 12 ), and a completion count of 1, indicating that the base tile210D of the second synchronous domain 538 will be waiting to start thethread on the second synchronous domain 538 until it has received onecompletion message.

Accordingly, when a tile 210 receives such data, such as tile 210E inFIG. 12 , it acknowledges that receipt by sending a completion message(with the thread ID (TID)) back to the base tile 210, here, base tile210D of the second synchronous domain 538. As part of the configurationprovided to the base tile 210 (at initial configuration or as part ofthe continue message), and stored in the completion table 422 (or pausetable 420) as a completion count, the base tile 210D knows how many suchcompletion messages the base tile 210 must receive in order to commenceexecution by the tiles 210 of the synchronous domain, in this case, thesecond synchronous domain 538. As completion messages are received bythe base tile 210, for the particular thread having that TID, thecompletion count of the pause table is decremented, and when it reacheszero for that thread, indicating all required completion messages havebeen received, the base tile 210 can commence execution of the thread.To commence execution, the TID of the thread is transferred to thecontinue queue 430, from which it is selected to run (at the appropriatespoke count for the appropriate time slice of the tile 210). It shouldbe noted that completion messages are not required for data which isdetermined during execution of the thread and which may be transferredbetween tiles 210 of the synchronous domain over the synchronous meshcommunication network 275.

There are several advantages to this type of thread control. This threadcontrol waits for all dependencies to be completed prior to starting thethread, allowing the started thread to have a fixed synchronousexecution time. The fixed execution time allows for the use of registerstages throughout the pipeline instead of FIFOs. In addition, while onethread of a tile 210 may be waiting to execute, other threads may beexecuting on that tile 210, providing for a much higher overallthroughput, and minimizing idle time and minimizing unused resources.

Similar control is provided when spanning synchronous domains, such asfor performance of multiple threads (e.g., for related compute threadsforming a compute fiber). For example, a first synchronous domain willinform the base tile 210 of the next synchronous domain, in acontinuation message transmitted over the asynchronous packet network265, how many completion messages it will need to receive in order forit to begin execution of the next thread. Also for example, foriterative looping spanning synchronous domains, a first synchronousdomain will inform the base tile 210 of the next synchronous domain, ina loop message (having a loop count and the same TID) transmitted overthe asynchronous packet network 265, how many completion messages itwill need to receive in order for it to begin execution of the nextthread.

It should also be mentioned that various delays may need to beimplemented, such as when first data is available from a first tile 210,while second data is still being determined by a second tile 210, bothof which will be needed in a next calculation by a third tile 210. Forsuch cases, delay may be introduced either at the output registers 380at the first tile 210 which created the first data, or in a tile memory325 of the third tile 210. This delay mechanism is also applicable todata which may be transferred from a first tile 210, using a second tile210 as a pass-through, to a third tile 210.

The pause table 420 is used to hold or pause the creation of a newsynchronous thread in the tile 210 until all required completionmessages have been received. A thread from a previous synchronous domainsends a message to a base tile 210 that contains the number ofcompletion messages to expect for the new synchronous thread, and theaction to take when all of the completion messages have been received.The actions include: call, continue, or loop. Many pause operations aretypically active concurrently. All messages for a specific pauseoperation (i.e., a set of pause and completion messages) will have thesame pause index within the respective messages. The pause index is theTID from the sending tile 210. Pause table 420 entries are initializedto be inactive with a completion delta count of zero. Receiving a pausemessage increments the delta count by the number of required completioncounts, and sets the pause table 420 entry to active. Receiving acompletion message decrements the delta count by one. It should be notedthat a completion message may arrive prior to the associated pausemessage, resulting in the delta count being negative. When a pause table420 entry is active with a delta count of zero then the associatedactivity (e.g., the new thread) is initiated (and the pause table 420entry is de-activated).

The continuation (or call) queue 430 holds threads ready to be startedon a synchronous domain. A thread is pushed into the continuation queue430 when all completions for a call operation are received. It should benoted that threads in the continuation queue 430 may require a TIDand/or XID to be allocated before the thread can be started on asynchronous domain, e.g., if all TIDs are in use, the threads in thecontinuation queue 430 can be started once a TID is freed and availablei.e., the thread may be waiting until TID and/or XIDs are available.

The reenter queue 445 holds threads ready to be started on a synchronousdomain, with execution of those threads having priority over those inthe continuation queue 430. A thread is pushed into the reenter queue445 when all completions for a continue operation are received, and thethread already has a TID. It should be noted that that threads in thereenter queue 445 cannot require allocation of a TID. Separate reenterand continue (or continuation) queues 445, 430 are provided to avoid adeadlock situation. A special type of continue operation is a loop. Aloop message contains a loop iteration count. The count is used tospecify how many times a thread is to be started once the pauseoperation completes.

An optional priority queue 425 may also be implemented, such that anythread having a thread identifier in the priority queue 425 is executedprior to execution of any thread having a thread identifier in thecontinuation queue 430 or in the reenter queue 445.

An iteration index 460 state is used when starting threads for a loopoperation. The iteration index 460 is initialized to zero andincremented for each thread start. The iteration index 460 is pushedinto the run queue 440 with the thread information from the continuequeue 430. The iteration index 460 is available as a selection to thedata path input multiplexer 365 within the first tile (base tile) 210 ofthe synchronous domain.

The loop iteration count 465 is received as part of a loop message,saved in the pause table 420, pushed into the continue queue 430, andthen used to determine when the appropriate number of threads have beenstarted for a loop operation.

The run queue 440 holds ready-to-run threads that have assigned TIDsand/or XIDs and can execute when the appropriate spoke count clocks haveoccurred. The TID pool 410 provides unique thread identifiers (TIDs) tothreads as they are started on the synchronous domain. Only threadswithin the continuation queue 430 can acquire a TID. The XID pool 415provides unique transfer identifiers (XIDs) to threads as they arestarted on the synchronous domain. Threads from the continue queue 430can acquire an XID. An allocated XID becomes the XID_WR for the startedthread.

For any given or selected program to be executed, the code orinstructions for that program, written or generated in any appropriateor selected programming language, are compiled for and loaded into thesystem 100, including instructions for the HTP 300 and HTF circuits 200,and any which may be applicable to the host processor 110, to providethe selected configuration to the system 100. As result, various sets ofinstructions for one or more selected computations are loaded into theinstruction RAMs 315 and the spoke RAMs 320 of each tile 210, and loadedinto any of the various registers maintained in the memory interfaces215 and HTF dispatch interface 225 of each tile 210, providing theconfigurations for the HTF circuits 200, and depending upon the program,also loaded into the HTP 300.

A kernel is started with a work descriptor message that contains zero ormore arguments, typically generated by the host processor 110 or the HTP300, for performance by one or more HTF circuits 200, for example andwithout limitation. The arguments are sent within the work descriptor AFmessage to the HTF dispatch interface 225. These arguments providethread-specific input values. A host processor 110 or HTP 300, using itsrespective operating system (“OS”) can send a “host” message to a kernelthat initializes a tile memory 325 location, with such host messagesproviding non-thread specific values. A typical example is a hostmessage that sends the base address for a data structure that is used byall kernel threads.

A host message that is sent to a kernel is sent to all HTF circuitclusters 205 where that kernel is loaded. Further, the order of sendinghost messages and sending kernel dispatches is maintained. Sending ahost message essentially idles that kernel prior to sending the message.Completion messages ensure that the tile memory 325 writes havecompleted prior to starting new synchronous threads.

The control messaging over the asynchronous packet network 265 is asfollows:

(1) The HTF dispatch interface 225 receives the host message and sendsan AF Data message to the destination tile 210. The destination tile 210writes the selected memory with the data of the AF Data message.

(2) The destination tile 210 sends an AF Complete message to the HTFdispatch interface 225 acknowledging that the tile write is complete.

(3) The HTF dispatch interface 225 holds all new kernel thread startsuntil all message writes have been acknowledged. Once acknowledged, theHTF dispatch interface 225 transmits an AF Call message to the base tileof the synchronous domain to start a thread.

The HTF dispatch interface 225 is responsible for managing the HTFcircuit cluster 205, including: (1) interactions with system 100software to prepare the HTF circuit cluster 205 for usage by a process;(2) dispatching work to the tiles 210 of the HTF circuit cluster 205,including loading the HTF circuit cluster 205 with one or more kernelconfigurations; (3) saving and restoring contexts of the HTF circuitcluster 205 to memory 125 for breakpoints and exceptions. As mentionedabove, the registers 475 of the HTF dispatch interface 225 may include awide variety of tables to track what has been dispatched to and receivedfrom any of the various tiles 210, such as tracking any of the messagingutilized in representative embodiments. The HTF dispatch interface 225primitive operations utilized to perform these operations are listed inTable 7.

TABLE 7 Primitive Operation Location Initiated by Operation DescriptionHTF Cluster Load HTF Application A HTF circuit cluster 205 checks eachKernel Dispatch received work descriptor to determine if theConfiguration Interface required kernel configuration matches thecurrently loaded configuration. If the work descriptor’s kernelconfiguration does not match the currently loaded configuration, thenthe HTF circuit cluster 205 waits for all previous work to complete andloads the new kernel configuration. Each work descriptor has the virtualaddress for the required kernel configuration. HTF Cluster Reset HTF OSReset all state within a HTF circuit cluster Dispatch 205 to allow a newkernel configuration or Interface kernel context to be loaded. HTFCluster Store HTF OS An HTF circuit cluster 205 can pause ContextDispatch execution due to an exception or breakpoint. Interface The HTFcircuit cluster 205 sends an interrupt to the OS to inform it of theevent. The OS determines if process context must be stored to memory fordebugger access. If process context is required, then the OS initiatesthe operation by interacting with the dispatch interface 225 of the HTFcircuit cluster 205. HTF Cluster Load HTF OS The context for an HTFcircuit cluster 205 can Context Dispatch be loaded from memory inpreparation to Interface resume execution. The OS initiates theoperation by interacting with the dispatch interface 225 of the HTFcircuit cluster 205. HTF Cluster Pause HTF OS The OS may need to pauseexecution on Dispatch running HTF circuit clusters 205 when theInterface owning process needs to be stopped. The process may need to bestopped if an exception or breakpoint occurred by a processor ordifferent HTF circuit cluster 205, or the process received a LinuxSignal. The OS initiates the pause by interacting with the dispatchinterface 225. HTF Cluster HTF OS Execution of a paused HTF circuitcluster 205 Resume Dispatch can be resumed by removal of the pauseInterface signal of the HTF circuit cluster 205. The OS initiates theresume by interacting with the dispatch interface 225 of the HTF circuitcluster 205. HTF Cluster Is Idle HTF OS The OS may need to determinewhen an HTF Dispatch circuit cluster 205 is idle and ready to accept aInterface new operation. The dispatch interface 225 has a number ofstate machines that perform various commands. These commands includecontext load, context store, pause, and configuration load. The OS mustensure that an HTF circuit cluster 205 is idle prior to issuing acommand.

FIGS. 16 and 17 provide an example of messaging and thread controlwithin a system 100, with an example computation provided to show howthe synchronous mesh communication network 275 and asynchronous packetnetwork 265 work together to execute a simple kernel, here, solving thesimple expression R=*A+B. To show such messaging, the computation hasbeen divided across two different synchronous domains 526 and 538. Thevariable B is passed as a host message to all HTF circuit clusters 205,and the address of A is passed as an argument to the call in the workdescriptor packet. The result R is passed back in the return data packetover the first interconnection network 150. The example does almost nocompute so the number of messages per compute performed is very high.The HTF circuits 200 have much higher performance when significantcomputation is performed within a loop such that the number of messagesper compute is low.

FIG. 16 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) 210 forming synchronous domainsand representative asynchronous packet network messaging for performanceof a computation by a HTF circuit cluster 205. FIG. 17 is a flow chartof representative asynchronous packet network messaging and execution byhybrid threading fabric configurable computing circuits (tiles) forperformance of the computation of FIG. 16 by a HTF circuit cluster 205.To begin, the host processor 110 sends a message (504) to the all HTFcircuit clusters 205 within the node, step 506. The message is the valueof the variable B. The message is contained in a single data packet,typically referred to as a work descriptor packet, that is written to adispatch queue 105 of the HIF 115 (illustrated in FIGS. 1 and 2 )associated with the process. The HIF 115 reads the message from thedispatch queue 105 and sends a copy of the packet to each HTF circuitcluster 205 assigned to the process. The dispatch interface 225 of theassigned HTF circuit cluster 205 receives the packet. It should also benoted that the HIF 115 performs various load balancing functions acrossall HTP 300 and HTF 200 resources.

The host processor 110 sends a call message (508) to one HTF circuitcluster 205 assigned to the process, step 510. The host processor 110can either manually target a specific HTF circuit cluster 205 to executethe kernel, or allow the HTF circuit cluster 205 to be automaticallyselected. The host processor 110 writes the call parameters to thedispatch queue associated with the process. The call parameters includethe kernel address, starting instruction, and the single argument(address of variable A). The host interface (HIF) 115 reads the queuedmessage and forwards the message as data packet on the firstinterconnection network 150 to the assigned HTF circuit cluster 205,typically the HTF circuit cluster 205 with the least load.

The HTF dispatch interface 225 receives the host message (value ofvariable B), waits until all previous calls to the HTF circuit cluster205 have completed, and sends the value to a first selected, destinationtile 210H using an AF message (512) over the asynchronous packet network265, step 514. The HTF dispatch interface 225 has a table ofinformation, stored in registers 475, for each possible host messagethat indicates the destination tile 210H, tile memory 325 and memoryregion (in RAM 405) for that tile 210H. The tile 210H uses the messageinformation to write the value to a memory 325 in the tile 210H, andonce the value is written to tile memory, then a write completion AFmessage (516) is sent via the asynchronous packet network 265 back tothe HTF dispatch interface 225, step 518.

The HTF dispatch interface 225 waits for all message completion messagesto arrive (in this case just a single message). Once all completionmessages have arrived, then the HTF dispatch interface 225 sends thecall argument (address of variable A) in an AF message (520) to a secondselected destination tile 210B for the value to be written into tilememory 325, step 522. The HTF dispatch interface 225 has a callarguments table stored in registers 475 that indicates the destinationtile 210B, tile memory 325 and memory region (in RAM 405) for that tile210B.

The HTF dispatch interface 225 next sends an AF call message (524) tothe base tile 210A of the first synchronous domain 526, step 528. The AFcall message indicates that a single completion message should bereceived before the call can start execution through the synchronoustile 210 pipeline. The required completion message has not arrived sothe call is paused.

Once the value is written to the tile memory 325 for that tile 210B,then a write completion message (530) is sent by the tile 210B via theasynchronous packet network 265 to the base tile 210A of the firstsynchronous domain 526, step 532.

The base tile 210A has received both the call message (524) and therequired completion message (530), and is now ready to initiateexecution on the synchronous domain 526 (tile pipeline). The base tile210A initiates execution by providing the initial instruction and avalid signal (534) to the tile 210B, via the synchronous meshcommunication network 275, step 536. The base tile 210A allocates an XIDvalue from an XID pool 415 for use in the first synchronous domain 526.If the XID pool 415 is empty, then the base tile 210A must wait to startthe synchronous pipeline until an XID is available.

As execution proceeds, the tile 210B or another tile 210E within thefirst synchronous domain 526 sends an AF continue message (540) to thebase tile 210C of a second synchronous domain 538, step 542. Thecontinue message contains the number of required completion messagesthat must arrive before the second synchronous domain 538 can initiateexecution (in this case a single completion message). The continuemessage also includes the transfer ID (XID). The XID is used as a writeindex in one synchronous domain (526), and then as a read index in thenext synchronous domain (538). The XID provides a common tile memoryindex from one synchronous domain to the next.

The tile 210B or another tile 210F within the first synchronous domain526 sends an AF memory load message (544) to the memory interface 215 ofthe HTF circuit cluster 205, step 546. The message contains a requestID, a virtual address, and the XID to be used as the index for writingthe load data to a destination tile (210G) memory 325.

The memory interface 215 receives the AF load message and translates thevirtual address to a node local physical address or a remote virtualaddress. The memory interface 215 uses the AF message's request ID toindex into a request table stored in registers 485 containing parametersfor the memory request. The memory interface 215 issues a load memoryrequest packet (548) for the first interconnection network 150 with thetranslated address and size information from the request table, step550.

The memory interface 215 subsequently receives a memory response packet(552) over the first interconnection network 150 with the load data(value for variable A), step 554. The memory interface 215 sends an AFmessage (556) to a tile 210G within the second synchronous domain 538,step 558. The AF message contains the value for variable A and the valueis written to tile memory using a parameter from the request tablestored in registers 485.

Once the value is written to tile memory, then an AF write completionmessage (560) is sent via the asynchronous packet network 265 to thebase tile 210C of the second synchronous domain 538, step 562.

The base tile 210C of the second synchronous domain 538 receives boththe continue message (540) and the required completion message (560) andis ready to initiate execution on the second synchronous domain 538(tile pipeline). The base tile 210C initiates execution by providing theinitial instruction and a valid signal (564) to a tile 210 of the secondsynchronous domain 538, step 566, such as tile 210H. The base tile 210Calso allocates an XID value from an XID pool for use in the secondsynchronous domain 538.

A tile 210H within the second synchronous domain performs the addoperation of the B value passed in from a host message and the A valueread from system memory 125, step 568. The resulting value is the Rvalue of the expression.

A tile 210J within the second synchronous domain sends an AF message(570) containing the R value to the HTF dispatch interface 225, step572. The AF message contains the allocated XID value from the base tile210A. The XID value is used as an index within the HTF dispatchinterface 225 for a table stored in registers 475 that hold returnparameters until the values have been read and a return messagegenerated for transmission over the first interconnection network 150.

An AF message (574) from the second synchronous domain (tile 210K) sendsthe XID value allocated in the first synchronous domain back to the basetile 210A to be returned to the XID pool, step 576. A firstinterconnection network 150 message (578) from the HTF dispatchinterface 225 is sent to the HIF 115, step 580. The HIF writes thereturn work descriptor to the dispatch return queue. Once the firstinterconnection network 150 has sent the return packet, the XID value issent in an AF message (582) by the HTF dispatch interface 225 to thebase tile 210C of the second synchronous domain 538 to be returned tothe XID pool, step 584.

It should be noted that in this example of FIGS. 16 and 17 , many tiles210 have been utilized in order to show a wide variety of AF messageswhich may be utilized for thread control. In practice, especially forsuch a simple computation, appreciably fewer tiles 210 can be utilized,such as to perform the computation completely within a singlesynchronous domain.

Another messaging example is provided for thread control across multiplesynchronous domains in FIGS. 18 and 19 , also using AF complete andcontinue messages on the asynchronous packet network 265. FIG. 18 is adiagram of representative hybrid threading fabric configurable computingcircuits (tiles) forming synchronous domains and representativeasynchronous packet network messaging for performance of a computationby a hybrid threading fabric circuit cluster. FIG. 19 is a flow chart ofrepresentative asynchronous packet network messaging and execution byhybrid threading fabric configurable computing circuits (tiles) forperformance of the computation of FIG. 18 by a hybrid threading fabriccircuit cluster.

For this example, the HTF dispatch interface 225 sends a message to thebase tile 210A of the first synchronous domain 526. The message starts athread on the first synchronous domain 526. The thread sends a threadcontinue message to a second synchronous domain 538. The continuemessage indicates that a thread is to be started on the secondsynchronous domain 538 when the specified number of completion messageshave been received. The first synchronous domain 526 sends a completionmessage to the second synchronous domain 538 causing the pause tocomplete and start the synchronous, second thread. The second threadsends a complete message back to the HTF dispatch interface 225indicating that the second synchronous thread completed, completing thedispatched kernel. Additional messages are shown in FIG. 18 that freeTID and XID identifiers.

The HTF dispatch interface 225 has received a work descriptor packet(602), has ensured that the correct kernel configuration is loaded, hasdetermined that the XID and TID pools are non-empty, obtaining the XIDand TID values for a new work thread from TID and XID pools stored inregisters 475 within the HTF dispatch interface 225, step 604. The HTFdispatch interface 225 starts kernel execution by sending an AF Callmessage (606) (with the allocated XID and TID values, e.g., XID=3 and(first type) TID=11) to the base tile 210A of a first synchronous domain526, step 608. The base tile 210A receives the AF Call message (606),determines that the TID and XID pools (410, 415) are non-empty,allocates TID and XID values (e.g., XID_WR=7 and (second type) TID=1),and that the Spoke Ram 320 is selecting the Base Tile as the input forthe tile data path, step 610, so that it begins execution with a firstdesignated instruction designated by the instruction index held in itsSpoke Ram 320, rather than potentially executing an instruction from aninstruction index which may have been provided by a previous tile 210(e.g., as discussed in greater detail below regarding conditionalexecution).

The base tile 210A starts a first thread (612) through the firstsynchronous domain 526 with the assigned TID value (e.g., (second type)TID=1), XID_RD assigned the value from the AF Call message (606) (e.g.,XID_RD=3), XID_WR assigned the value obtained from the XID pool (e.g.,XID_WR=7), and TID from the AF Call message (606) (e.g., (first type)TID=11), step 614.

As the computation proceeds in the first synchronous domain 526, anothertile 210B within the first synchronous domain 526 sends an AF Continuemessage (616) to the base tile 210D of the second synchronous domain538, step 618. The AF Continue message (616) provides the informationnecessary to start a second thread on the second synchronous domain 538when the appropriate number of completion messages have arrived. The AFContinue message (616) includes a completion count field having a valuethat specifies the number of required completion messages. One of thetiles (210C) in the first synchronous domain 526 also transmits a freeXID (e.g., XID=3) message (641) to the HTF dispatch interface 225.

The AF Continue message (616) can include either the TID or XID_WR valueas the index into the pause table 420 on the destination base tile 210D.The pause table accumulates the received completion messages anddetermines when the requisite number have arrived and a new thread canbe started, step 620. The tile 210B that sends the AF Continue message(616) sends the selected TID or XID_WR value as the PID (pause tableindex) and changes the synchronous domain's downstream TID value to theselected value (e.g., (first type) TID=11, (second type) TID=1). Thisnew TID value is passed in all AF completion messages to be used as theindex into the pause table 420 of the base tile 210D.

An AF Complete message (622) is sent to the base tile 210D of the secondsynchronous domain 538 with the TID value (e.g., (first type) TID=11),step 624. The AF Complete message (622) decrements the entry of a deltacount field in the pause table 420 of the base tile 210D. The AFComplete message (622) and the AF Continue message (616) could arrive inany order. The last message to arrive will observe that the AF Continuemessage (616) has arrived and the delta count field in the pause table420 has reached zero. This condition indicates that the pause hascompleted and a second synchronous thread (626) can be started. The basetile 210D also determines or observes that the pause operation hascompleted, determines that the XID identifier pool is non-empty andallocates an XID (e.g., XID=5), and that the Spoke Ram is selecting theBase Tile as the input for the tile data path, step 628.

The base tile 210D then starts the second synchronous thread (626)through the second synchronous domain 538, step 630, with TID and XID_RDassigned the value obtained from the AF Continue message (616) (e.g.,(first type) TID=11, (second type) TID=1, XID_RD=7). The XID_WR isassigned the value obtained from the XID pool in step 628 (e.g.,XID_WR=5).

When the computations of the second synchronous thread (626) havecompleted, several housekeeping messages are sent by the various tiles210 of the second synchronous domain 538. An AF Free TID message (632)is sent to the base tile 210A of the first synchronous domain 526, step634, and the receiving base tile 210A adds the TID value to the TID pool410, step 636, so it is available once again for use. An AF Free XIDmessage (638) is sent to the base tile 210A of the first synchronousdomain 526, step 640, and the receiving base tile 210 adds the XID valueto the XID pool 415, step 642, also so it is available once again foruse. An AF Complete message (644) is sent to the HTF dispatch interface225 indicating that the second synchronous thread 626 has completed,step 646. The HTF dispatch interface 225 has a count of expectedcompletion messages. The AF Complete message (644) includes the XID_WRvalue and TID value ((first type) TID=11) of the second synchronousdomain 538 to the dispatch interface. The HTF dispatch interface 225then sends an AF Free XID message (648) to the base tile 210D of thesecond synchronous domain 538, step 650. The receiving base tile 210Dthen adds the XID value to the XID pool 415, step 652, so it isavailable once again for use.

A data transfer operation is used to transfer data from one synchronousdomain to the next. Typically, a data transfer is used in conjunctionwith a load operation obtaining data from memory 125. Calculated datafrom the first synchronous domain 526 is needed in the secondsynchronous domain 538 once the load data has arrived at the secondsynchronous domain 538. In this case, a single pause is sent from thefirst synchronous domain 526 to the second synchronous domain 538 thatcontains the total count of completion messages from all load and datatransfer operations.

The data transfer operation between synchronous domains then utilizes avariation of step 624. Instead of sending an AF Complete message (622)in step 624, the first synchronous domain 526 sends an AF Data messageto the second synchronous domain 538 with data. The destination tile 210in the second synchronous domain 538 writes the data within the AF Datamessage to the selected tile memory 325. The tile 210 that receives theAF Data message then sends an AF Complete message to the base tile 210of the second synchronous domain 538. The base tile 210 of the secondsynchronous domain 538 may then launch the second thread on the secondsynchronous domain 538 once the load data has arrived at the secondsynchronous domain 538.

Control over iterative thread looping across synchronous domainsutilizes a similar control messaging schema. The loop message flowallows multiple synchronous domain starts from a single loop message.Each of the started synchronous threads is able to access its iterationindex. FIG. 20 is a diagram of representative hybrid threading fabricconfigurable computing circuits (tiles) forming synchronous domains andrepresentative asynchronous packet network messaging for performance ofa loop in a computation by a hybrid threading fabric circuit cluster.FIG. 21 is a flow chart of representative asynchronous packet networkmessaging and execution by hybrid threading fabric configurablecomputing circuits (tiles) for performance of the loop in a computationof FIG. 20 by a hybrid threading fabric circuit cluster.

FIG. 20 shows three synchronous domains, first synchronous domain 526,second synchronous domain 538, and third synchronous domain 654. Thefirst synchronous domain 526 is used for pre-loop setup, the secondsynchronous domain 538 is started with an iteration count (IterCnt) forthe number of threads, and the final, third synchronous domain 654 ispost-loop. It should be noted that loops can be nested, as well, usingadditional layers of indexing, discussed in greater detail below.

Referring again to FIG. 11 , the control registers 340 include acompletion table 422 (or pause table 420). For loops, two kinds ofcompletion information are maintained in the completion table 422, afirst completion count pertaining to the number of completion messageswhich should arrive before a thread may start, as discussed above, and asecond, loop or iteration (completion) count, to track how many loopthreads have been started and completed. A loop is started by sending anAF loop message containing a loop count (and various TIDs, discussedbelow) to the base tile 210 of a synchronous domain. The loop count isstored in the completion table 422 (or pause table 420), and is used todetermine the number of times a new thread is started on the synchronousdomain. In one embodiment, each thread is started with a new TIDobtained from the TID pool 410. Each active thread has a unique TIDallowing thread private variables, for example. The threads of nestedloops are provided with access to the data or variables of its own TID,plus the TIDs of the outer loops. In a second embodiment discussedbelow, TIDs are re-used by successive threads of the loop.

TIDs are returned to the TID pool 410 by an AF message being sent from atile within a synchronous domain when the thread is terminating, whichmay be either an AF Complete message, or for the second embodiment, anAF reenter message. This can also be accomplished by a Free TID messageto the base tile 210. The AF message that returns the TID to the pool orre-uses the TID also is used by the loop base tile 210 to maintain acount of the number of active loop threads in the loop count of thecompletion table 422 (or pause table 420). When the number of activeloop threads reaches zero, then the loop is complete. When the loopcompletion is detected by the loop count going to zero, then an AFComplete message is sent to the post-loop synchronous domain informingof the completion. This mechanism provides for minimal (if not zero)idle cycles for nested loops, resulting in better performance.

Referring to FIGS. 20 and 21 , the first synchronous domain 526(illustrated as tile 210B, although it can be from any other tile in thefirst synchronous domain 526) sends an AF Continue message (656) to thebase tile 210D of the third, post-loop synchronous domain 654, step 658,to wait for the loop completion message (which will be from the secondsynchronous domain 538). One of the tiles in the first synchronousdomain 526, illustrated as tile 210B, sends an AF Loop message (660)with the iteration (loop) count to the base tile 210C of the loopdomain, which is the second synchronous domain 538, step 664. The basetile 210C starts the loop (IterCnt) threads (662, e.g., 662 ₀, 662 ₁,through 662 _(N−1), where “N” is the iteration count (IterCnt)) on thesecond synchronous domain 538. Each thread 662 has the same TID andXID_RD identifiers. The XID_WR identifier is allocated by the loop basetile 210C if enabled. The iteration index (i.e., ordered from zero toIterCnt-1 (N−1)) is accessible as a data path multiplexer selection inthe base tile 210C of the loop domain.

Each iteration of the loop domain then sends an AF Complete message(666) back to the base tile 210C of the second synchronous (loop) domain538, step 668. It should be noted that the second synchronous domain 538shown in FIG. 20 may actually be several synchronous domains. For thecase in which multiple synchronous domains form the loop, the threads ofthe last synchronous domain of the loop should transmit the AF Completemessages (666), so that the post-loop third synchronous domain 654properly waits for all loop operations to complete. Once the base tile210C of the second synchronous (loop) domain 538 has received alliteration AF Complete messages (666), it then sends a loop AF Completemessage (or an AF continuation message) (670) to the base tile 210D ofthe third (post-loop) synchronous domain 654.

For looping, including nested and doubly-nested looping, severaladditional and novel features are utilized, such as to minimize idletime, including a reenter queue 445 and additional sub-TIDs, such as aTID₂ for the outermost loop, a TID₁ for the middle or intermediate loop,and a TID₀ for the innermost loop, for example and without limitation.Each thread that is executing in the loop than also has a unique TID,such as TID₂s 0-49 for an outer loop which will have fifty iterations,which are also utilized in the corresponding completion messages wheneach iteration completes execution, also for example and withoutlimitation.

Referring again to FIG. 11 , several novel mechanisms are provided tosupport efficient looping, and to minimize idle time. For example, loopswith data-dependent end conditions (e.g., “while” loops) require thatthe end condition to be calculated as the loop is executed. Also forcontrol and execution of looping, a potential deadlock issue may ariseif all TIDs are allocated from the TID pool 410, but the thread at thehead of a queue for execution is a new loop, which cannot executebecause of a lack of available TIDs, blocking other looping threadswhich cannot complete and free up their assigned TIDs. Accordingly, inrepresentative embodiments, control registers 340 include two separatequeues for ready-to-run threads, with a first queue for initiating newloops (the continuation queue 430, also utilized for non-loopingthreads), and a second, separate queue (the reenter queue 445) for loopcontinuation. The continuation queue 430 allocates a TID from the TIDpool 410 to start a thread, as previously discussed. The reenter queue445 uses the previously allocated TID, as each iteration of a loopthread executes and transmits an AF reenter message with the previouslyallocated TID. Any thread (TID) in the reenter queue 445 will be movedinto the run queue 440 ahead of the threads (TIDs) which may be in theother queues (continuation queue 430). As a result, once a loop islaunched, loop iteration occurs very rapidly, with each next thread ofthe loop being launched rapidly through use of the separate reenterqueue 445, and further, without the potential for deadlock issues. Inaddition, the reenter queue 445 allows this rapid execution that is verysignificant for loops with data-dependent end conditions, which can nowrun effectively without interruption to the last iteration whichproduces the data-dependent end conditions.

Referring again to FIGS. 9 and 10 , control registers 340 include amemory region RAM 405. The memory region RAM 405, in variousembodiments, may also maintain a top of TID stack (with identifiers)through nested loops, described below. As mentioned above, each nestedloop initiates threads with a new (or re-used) set of TIDs. Threads of aloop may need to have access to its TID plus the TIDs of the outer loopthreads. Having access to the TIDs of each nested loop thread allowsaccess to each thread's private variables, such as the different levelor types of TIDs described above, TID₀, TID₁ and TID₂. The top of stackTID identifier indicates the TID for the active thread. The top of stackTID identifier is used to select which of the three TIDs (TID₀, TID₁ andTID₂) is used for various operations. These three TIDs and the top ofstack TID identifier are included in synchronous fabric controlinformation (or messages) transmitted on the synchronous meshcommunication network 275, so are known to each thread. Because multipleTIDs are included within a synchronous fabric message and include a topof stack TID identifier, the multiple TIDs allow a thread in a nestedloop to access variables from any level within the nested loop threads.The selected TID plus a tile memory region RAM 405 identifier is used toaccess a private thread variable.

Another novel feature of the present disclosure is the mechanism toorder loop thread execution to handle loop iteration dependencies, whichalso accommodates any delays in completion messages and data receivedover the asynchronous packet network 265. FIG. 23 is a diagram of tiles210 forming synchronous domains and representative asynchronous packetnetwork messaging and synchronous messaging for performance of a loop ina computation by a hybrid threading fabric circuit cluster. Asillustrated in FIG. 23 , multiple synchronous domains 682, 684, and 686,are involved in performance of a loop computation, a second synchronousdomain 682, a third synchronous domain 684, a fourth synchronous domain686, in addition to the pre-loop first synchronous domain 526 andpost-loop (fifth) synchronous domain 654. The loop computation may beany kind of loop, including nested loops, and in this case, there aredata dependencies within the various loops. For example, these datadependencies may occur within a single iteration, such as wheninformation is needed from memory 125, involving AF messaging over theasynchronous packet network 265. As a result, thread execution shouldproceed in a defined order, and not merely whenever any particularthread has a completion count of zero (meaning that thread is notwaiting on any data, with all completion messages for that thread havingarrived).

To provide ordered loop thread execution, in representative embodiments,additional messaging and additional fields are utilized in thecompletion table 422, for each loop iteration. The loop base tile 210Bprovides four pieces of information (for each loop iteration) that ispassed in synchronous messages 688 through each synchronous domain 682,684, 686 through the synchronous mesh communication network 275 (i.e.,passed to every successive tile 210 in that given synchronous domain),and AF continue messages 692 to the base tiles 210 of successivesynchronous domains via the asynchronous packet network 265 (which isthen passed in synchronous messages to each successive tile 210 in thatgiven synchronous domain). Those four fields of information are thenstored and indexed in the completion table 422 and utilized forcomparisons as the loop execution progresses. The four pieces ofinformation are: a first flag indicating the first thread of a set ofthreads for a loop, a second flag indicating the last thread of a set ofthreads for a loop, the TID for the current thread, and the TID for thenext thread. The TID for the current thread is obtained from a pool ofTIDs, and the TID for the next thread is the TID from the pool that willbe provided for the next thread. These four pieces of information areused by the base tile of each successive synchronous domain to orderthread starts. A thread can be started if its dependency count hasreached zero and the thread is the first thread for a loop, or thethread TID equals the next TID for the previously started thread.

Stated another way, the thread control circuitry 330 (which generallyincludes various state machines) checks the completion table 422, foreach thread which has received all data completions (so would otherwisebe ready-to-run), whether that thread is the next thread to run (havingthe next thread ID, e.g., TID=4), and if so, moves that thread (TID=4)into the run queue 440, and if not, does not start that thread (e.g., athread whose data completion count went to zero but has a TID=5) butmaintains the index of which TID is next to start. When the datacompletion for the thread having the next TID decrements to zero (TID=4in this case), so all completion messages have arrived, that thread isqueued for execution, and that thread (TID=4) which will be executinghas a next TID as well, in this case, its next TID is TID=5.Accordingly, when the thread having TID=4 has completed, the threadcontrol circuitry 330 checks the completion table 422 and now determinesthat the thread (TID=5) is the next thread ID, and queues that threadfor execution. When the thread ID is then the last TID, following itsexecution, an AF completion message (656) can be transmitted to thepost-loop base tile (in this case, 210E). It should be noted that thisuse of the additional fields in the completion table 422 may be extendedto any situation in which a particular ordering of thread executionshould be maintained.

FIG. 24 is a block and circuit diagram of a representative embodiment ofconditional branching circuitry 370. A synchronous domain, such as thefirst, second and third synchronous domains mentioned above, is a set ofinterconnected tiles, connected in a sequence or series through thesynchronous mesh communication network 275. Execution of a thread beginsat the first tile 210 of the synchronous domain, referred to as a basetile 210, and progresses from there via the configured connections ofthe synchronous mesh communication network 275 to the other tiles 210 ofthe synchronous domain. As illustrated in FIG. 24 , when a tile 210 hasbeen configured as a base tile 210 of the synchronous domain (as thoseconfigurations have been loaded into the HTF circuit 200 in advance ofrun time), the selection 374 of a configuration memory multiplexer 372is set equal to 1, which thereby selects the spoke RAM 320 to providethe instruction index for selection of instructions from the instructionRAM 315. For all other tiles 210 of the synchronous domain, theselection 374 of a configuration memory multiplexer 372 is set equal to0, which thereby selects an instruction index provided by the previoustile 210 in the sequence of tiles 210 of the synchronous domain. As aresult, the base tile 210 provides the instruction index (or theinstruction) to be executed to the next, second tile of the domain, viadesignated fields (or portions) of the communication lines (or wires)270B and 270A (which have been designated the master synchronous inputs,as mentioned above). By default, thereafter, this next tile 210, andeach succeeding tile 210 of the synchronous domain, will provide thesame instruction to each next tile 210 of the connected tiles 210 forexecution, as a static configuration.

In representative embodiments, however, a mechanism is provided fordynamic self-configuration, using the spoke RAM 320, the instruction RAM315, and the conditional branching circuitry 370. Referring to FIG. 24 ,for a current tile 210, the ALB Op 310 may be configured to generate anoutput which is the outcome of a test condition, such as whether oneinput is greater than a second input, for example. That test conditionoutput is provided to the conditional branching circuitry 370, oncommunication lines (or wires) 378. When the conditional branchingcircuitry 370 is enabled (through one or more bits of an instructionprovided on lines (or wires) 379), the test condition output is utilizedto select the next instruction index (or instruction) which is providedto the next tile 210 of the synchronous domain, such as to selectbetween “X” instruction or “Y” instruction for the next tile 210,providing conditional branching of the data path when the first or thesecond instruction is selected. Such conditional branching may also becascaded, such as when the next tile 210 is also enabled to provideconditional branching. By selecting the next instruction for one or moreof the next tiles 210, dynamic self-configuration andself-reconfiguration is enabled in each such HTF circuit cluster 205.

In a representative embodiment, the conditional branching circuitry 370has been arranged to select or toggle between two differentinstructions, depending on the test condition result. The branch enableis provided in one of the fields of the current (or currently next)instruction, and is provided to an AND gate 362 of the conditionalbranching circuitry 370, where it is ANDed with the test conditionoutput. Depending on the test condition output being a logical “0” or“1”, AND gate 362 will generate a logical “0” or “1” as an output, whichis provided as an input to OR gate 364. Another designated bit of aselected field of the currently next instruction index, typically theleast significant bit (“LSB”) of the next instruction index, is alsoprovided to the OR gate 364, where it is ORed with the output of the ANDgate 362. If the LSB of the next instruction index is a zero, and it isORed with a logical “1” of the output of the AND gate 362, then the nextinstruction index which is output has been incremented by one, providinga different next instruction index to the next tile 210. If the LSB ofthe next instruction index is a zero, and it is ORed with a logical “0”of the output of the AND gate 362, then the next instruction index whichis output has not been incremented by one, providing the same nextinstruction index to the next tile 210. As a result, the current tile210 has conditionally specified an alternate instruction for connectedtiles 210 to execute, enabling the performance of one or more casestatements in the HTF circuit cluster 205. The alternate instruction ischosen by having the current tile's data path produce a Booleanconditional value, and using the Boolean value to choose between thecurrent tile's instruction and the alternate instruction provided as thenext instruction index to the next tile 210 in the synchronous domain.Also a result, the current tile 210 has dynamically configured the nexttile 210, and so on, resulting in dynamic self-configuration andself-reconfiguration in each HTF circuit cluster 205.

IV. Hybrid Threading Processor 300:

FIG. 25 is a high-level block diagram of a representative embodiment ofa hybrid threading processor (“HTP”) 300. FIG. 26 is a detailed blockdiagram of a representative embodiment of a thread memory 720 (alsoreferred to as a thread control memory 720) of the HTP 300. FIG. 27 is adetailed block diagram of a representative embodiment of a networkresponse memory 725 of the HTP 300. FIG. 28 is a detailed block diagramof a representative embodiment of an HTP 300. FIG. 29 is a flow chart ofa representative embodiment of a method for self-scheduling and threadcontrol for an HTP 300.

An HTP 300 typically comprises one or more processor cores 705 which maybe any type of processor core, such as a RISC-V processor core, an ARMprocessor core, etc., all for example and without limitation. A corecontrol circuit 710 and a core control memory 715 are provided for eachprocessor core 705, and are illustrated in FIG. 25 for one processorcore 705. For example, when a plurality of processor cores 705 areimplemented, such as in one or more HTPs 300, corresponding pluralitiesof core control circuits 710 and core control memories 715 are alsoimplemented, with each core control circuit 710 and core control memory715 utilized in the control of a corresponding processor core 705. Inaddition, one or more of the HTPs 300 may also include data path controlcircuitry 795, which is utilized to control access sizes (e.g., memory125 load requests) over the first interconnection network 150 to managepotential congestion of the data path.

In turn, a core control circuit 710 comprises control logic and threadselection circuitry 730 and network interface circuitry 735. The corecontrol memory 715 comprises a plurality of registers or other memorycircuits, conceptually divided and referred to herein as thread memory(or thread control memory) 720 and network response memory 725. Thethread memory 720 includes a plurality of registers to store informationpertaining to thread state and execution, while the network responsememory 725 includes a plurality of registers to store informationpertaining to data packets transmitted to and from first memory 125 onthe first interconnection network 150, such as requests to the firstmemory 125 for reading or storing data, for example and withoutlimitation.

Referring to FIG. 26 , the thread memory 720 includes a plurality ofregisters, including thread ID pool registers 722 (storing apredetermined number of thread IDs which can be utilized, and typicallypopulated when the system 100 is configured, such as with identificationnumbers 0 to 31, for a total of 32 thread IDs, for example and withoutlimitation); thread state (table) registers 724 (storing threadinformation such as valid, idle, paused, waiting for instruction(s),first (normal) priority, second (low) priority, temporary changes topriority if resources are unavailable); program counter registers 726(e.g., storing an address or a virtual address for where the thread iscommencing next in the instruction cache 740); general-purpose registers728 for storing integer and floating point data; pending fiber returncount registers 732 (tracking the number of outstanding threads to bereturned to complete execution); return argument buffers 734 (“RAB”,such as a head RAB as the head of a link list with return argumentbuffers), thread return registers 736 (e.g., storing the return address,a call identifier, any thread identifier associated with the callingthread); custom atomic transaction identifier(s) registers 738; eventreceived mask registers 742 (to designate which events to “listen” for,as discussed in greater detail below), even state registers 744, and adata cache 746 (typically providing 4-8 cache lines of cache memory foreach thread). All of the various registers of the thread memory 720 areindexed using the assigned thread ID for a given or selected thread.

Referring to FIG. 27 , the network response memory 725 includes aplurality of registers, such as memory request (or command) registers748 (such as commands to read, write, or perform a custom atomicoperation); thread ID and transaction identifiers (“transaction IDs”)registers 752 (with transaction IDs utilized to track any requests tomemory, and associating each such transaction ID with the thread ID forthe thread which generated the request to memory 125); a request cacheline index register 754 (to designate which cache line in the data cache746 is to be written to when data is received from memory for a giventhread (thread ID), register bytes register 756 (designating the numberof bytes to write to the general-purpose registers 728); and ageneral-purpose register index and type registers 758 (indicating whichgeneral-purpose register 728 is to be written to, and whether it is signextended or floating point).

As described in greater detail below, an HTP 300 will receive a workdescriptor packet. In response, the HTP 300 will find an idle or emptycontext and initialize a context block, assigning a thread ID to thatthread of execution (referred to herein generally as a “thread”), if athread ID is available, and puts that thread ID in a an execution (i.e.,“ready-to-run”) queue 745. Threads in the execution (ready-to-run) queue745 are selected for execution, typically in a round-robin or “barrel”style selection process, with a single instruction for the first threadprovided to the execution pipeline 750 of the processor core 705,followed by a single instruction for the second thread provided to theexecution pipeline 750, followed by a single instruction for the thirdthread provided to the execution pipeline 750, followed by a singleinstruction for the next thread provided to the execution pipeline 750,and so on, until all threads in the execution (ready-to-run) queue 745have had a corresponding instruction provided to the execution pipeline750, at which point the thread selection commences again with a nextinstruction for the first thread in the execution (ready-to-run) queue745 provided to the execution pipeline 750, followed by a nextinstruction for the second thread provided to the execution pipeline750, and so on, cycling through all of the threads of the execution(ready-to-run) queue 745. This execution will continue for each suchthread until execution for that thread has been completed, such as byexecuting a thread return instruction, at which point a response packet(having the results of the thread execution) is transmitted back to thesource of the work descriptor packet, i.e., back to the source of thework descriptor call packet. In addition, in a representative embodimentand as discussed in greater detail below, the execution (ready-to-run)queue 745 is optionally provided with different levels of priority,illustrated as a first priority queue 755 and a second (lower) priorityqueue 760, with execution of the threads in the first priority queue 755occurring more frequently than the execution of the threads in thesecond (lower) priority queue 760.

As a result, the HTP 300 is an “event driven” processor, and willautomatically commence thread execution upon receipt of a workdescriptor packet (provided a thread ID is available, but without anyother requirements for initiating execution), i.e., arrival of a workdescriptor packet automatically triggers the start of thread executionlocally, without any reference to or additional requests to memory 125.This is tremendously valuable, as the response time to commenceexecution of many threads in parallel, such as thousands or threads, iscomparatively low. The HTP 300 will continue thread execution untilthread execution is complete, or it is waiting for a response, at whichpoint that thread will enter a “pause” state, as discussed in greaterdetail below. A number of different pause states are discussed ingreater detail below. Following receipt of that response, the thread isreturned to an active state, at which point the thread resumes executionwith its thread ID returned to the execution (ready-to-run) queue 745.This control of thread execution is performed in hardware, by thecontrol logic and thread selection circuitry 730, in conjunction withthread state information stored in the thread memory 720.

In addition to a host processor 110 generating work descriptor packets,an HTP 300 can also generate and transmit work descriptor packets toinitiate work, as one or more compute threads, on another computingresource, such as another HTP 300 or any HTF circuit 200. Such a workdescriptor packet is a “call” work descriptor packet, and generallycomprises a source identifier or address for the host processor 110 orthe HTP 300 which is generating the call work descriptor packet, athread ID (such as a 16-bit call identifier (ID)) used to identify orcorrelate the return with the original call, a 64-bit virtual kerneladdress (as a program count, to locate the first instruction to beginexecution of the thread, typically held in the instruction cache 740 ofan HTP 300 (or of a HTF circuit 200), which also may be a virtualaddress space), and one or more call arguments, e.g., up to four callarguments).

Similarly, when the thread has been completed, the HTP 300 or HTFcircuit 200 generates another work descriptor packet, referred to as a“return” work descriptor packet, which is generally created when the HTP300 or HTF circuit 200 executes the last instruction of the thread,referred to as a return instruction, with the return work descriptorpacket assembled by the packet encoder 780, discussed below. The returnpacket will be addressed back to the source (using the identifier oraddress provided in the call work descriptor packet), the thread ID (orcall ID) from the call work descriptor packet (to allow the source tocorrelate the return with the issued call, especially when multiplecalls have been generated by the source and are simultaneouslyoutstanding), and one or more return values (as results), such as up tofour return values.

FIG. 28 is a detailed block diagram of a representative embodiment of anHTP 300. For ease of illustration and discussion, it should be notedthat not all registers of the thread memory 720 and the network responsememory 725 are illustrated in FIG. 28 . Referring to FIG. 28 , the corecontrol circuit 710 comprises control logic and thread selectioncircuitry 730 and network interface circuitry 735. The control logic andthread selection circuitry 730 comprises circuitry formed usingcombinations of any of a plurality of various logic gates (e.g., NAND,NOR, AND, OR, EXCLUSIVE OR, etc.) and various state machine circuits(control logic circuit(s) 731, thread selection control circuitry 805),and multiplexers (e.g., input multiplexer 787, thread selectionmultiplexer 785), for example and without limitation. The networkinterface circuitry 735 includes AF input queues 765 to receive datapackets (including work descriptor packets) from the firstinterconnection network 150; AF output queues 770 to transfer datapackets (including work descriptor packets) to the first interconnectionnetwork 150; a data packet decoder circuit 775 to decode incoming datapackets from the first interconnection network 150, take data (indesignated fields) and transfer the data provided in the packet to therelevant registers of the thread memory 720 and the network responsememory 725 (in conjunction with the thread ID assigned to the thread bythe control logic and thread selection circuitry 730, as discussed ingreater detail below, which thread ID also provides or forms the indexto the thread memory 720; and data packet encoder circuit 780 to encodeoutgoing data packets (such as requests to memory 125, using atransaction ID from thread ID and transaction identifiers (“transactionIDs”) registers 752) for transmission on the first interconnectionnetwork 150. The data packet decoder circuit 775 and the data packetencoder circuit 780 may each be implemented as state machines or otherlogic circuitry. Depending upon the selected embodiment, there may be aseparate core control circuit 710 and separate core control memory 715for each HTP processor core 705, or a single core control circuit 710and single core control memory 715 may be utilized for multiple HTPprocessor cores 705.

When a work descriptor packet arrives, the control logic and threadselection circuitry 730 assigns an available thread ID to the thread ofthe word descriptor packet, from the thread ID pool registers 722, withthe assigned thread ID used as an index to the other registers of thethread memory 720 which are then populated with corresponding data fromthe work descriptor packet, typically the program count and one or morearguments. The control logic and thread selection circuitry 730initializes the remainder of the thread context state autonomously inpreparation for starting the thread executing instructions, such asloading the data cache registers 746 and loading the thread returnregisters 736, for example and without limitation. Also for example, anexecuting thread has main memory stack space and main memory contextspace. The context space is only used if the state of the thread needsto be written to memory to be accessed by the host. Each HTP 300processor core 705 is initialized with a core stack base address and acore context base address, where the base addresses point a block ofstacks and a block of context spaces. The thread stack base address isobtained by taking the core stack base address and adding the thread IDmultiplied by the thread stack size. The thread context base address isobtained in a similar fashion.

That thread ID is given a valid status (indicating it is ready toexecute), and the thread ID is pushed to the first priority queue 755 ofthe execution (ready-to-run) queue(s) 745, as threads are typicallyassigned a first (or normal) priority. Selection circuitry of thecontrol logic and thread selection circuitry 730, such as a multiplexer785, selects the next thread ID in the execution (ready-to-run) queue(s)745, which is used as in index into the thread memory 720 (the programcount registers 726 and thread state registers 724), to select theinstruction from the instruction cache 740 which is then provided to theexecution pipeline 750 for execution. The execution pipeline thenexecutes that instruction.

At completion of execution of the instruction, under the control of thecontrol logic and thread selection circuitry 730 the same triplet ofinformation (thread ID, valid state, and priority) can be returned tothe execution (ready-to-run) queue(s) 745, for continued selection forround-robin execution, depending upon various conditions. For example,if the last instruction for a selected thread ID was a returninstruction (indicating that thread execution was completed and a returndata packet is being provided), the control logic and thread selectioncircuitry 730 will return the thread ID to the available pool of threadIDs in the thread ID pool registers 722, to be available for use byanother, different thread. Also for example, the valid indicator couldchange, such as changing to a pause state (such as while the thread maybe waiting for information to be returned from or written to memory 125or waiting for another event), and in which case, the thread ID (nowhaving a pause status) is not returned to the execution (ready-to-run)queue(s) 745 until the status changes back to valid.

Continuing with the former example, when the last instruction for aselected thread ID was a return instruction, the return information(thread ID and return arguments) is then pushed by the executionpipeline 750 to the network command queue 790, which is typicallyimplemented as first-in, first out (FIFO). The thread ID is used as anindex into the thread return registers 736 to obtain the returninformation, such as the transaction ID and the source (caller) address(or other identifier), and the packet encoder circuit then generates anoutgoing return data packet (on the first interconnection network 150).

Continuing with the latter example, an instruction of a thread may be aload instruction, i.e., a read request to the memory 125, which is thenpushed by the execution pipeline 750 to the network command queue 790.The packet encoder circuit then generates an outgoing data packet (onthe first interconnection network 150) with the request to memory 125(as either a read or a write request), including the size of the requestand an assigned transaction ID (from the thread ID and transaction IDsregisters 752, which is also used as an index into the network responsememory 725), the address of the HTP 300 (as the return address of therequested information). When a packet is then received from the firstinterconnection network 150 and decoded, the transaction ID is used asan index into the network response memory 725, the thread ID of thethread which made the request is obtained, which also provides thelocation in the data cache 746 to write the data returned in theresponse, with the transaction ID then returned to the thread ID andtransaction ID registers 752 to be reused, and the status of thecorresponding thread ID is set again to valid and the thread ID is againpushed to the execution (ready-to-run) queue(s) 745, to resumeexecution.

A store request to memory 125 is executed similarly, with the outgoingpacket also having the data to be written to memory 125, an assignedtransaction ID, the source address of the HTP 300, and with the returnpacket being an acknowledgement with the transaction ID. The transactionID is also then returned to the thread ID and transaction ID registers752 to be reused, and the status of the corresponding thread ID is setagain to valid and the thread ID is again pushed to the execution(ready-to-run) queue(s) 745, to resume execution.

FIG. 29 is a flow chart of a representative embodiment of a method forself-scheduling and thread control for an HTP 300, and provides a usefulsummary, with the HTP 300 having already been populated withinstructions in the instruction cache 740 and a predetermined number ofthread IDs in the thread identifier pool register 722. The methodstarts, step 798, upon reception of a work descriptor packet. The workdescriptor packet is decoded, step 802, and the various registers of thethread memory 720 is populated with the information received in the workdescriptor packet, initializing a context block, step 804. When a threadID is available, step 806, a thread ID is assigned, step 808 (and if athread ID is not available in step 806, the thread will wait until athread ID becomes available, step 810). A valid status is initiallyassigned to the thread (along with any initially assigned priority, suchas a first or second priority), step 812, and the thread ID is providedto the execution (ready-to-run) queue 745, step 814. A thread ID in theexecution (ready-to-run) queue 745 is then selected for execution (at apredetermined frequency, discussed in greater detail below), step 816.Using the thread ID, the thread memory 720 is accessed, and a programcount (or address) is obtained, step 818. The instruction correspondingto the program count (or address) is obtained from the instruction cache740 and provided to the execution pipeline 750 for execution, step 820.

When the thread execution is complete, i.e., the instruction beingexecuted is a return instruction, step 822, the thread ID is returned tothe thread ID pool registers 722 for reuse by another thread, step 824,the thread memory 720 registers associated with that thread ID may becleared (optionally), step 826, and the thread control may end for thatthread, return step 834. When the thread execution is not complete instep 822, and when the thread state remains valid, step 828, the threadID (with its valid state and priority) is returned to the execution(ready-to-run) queue 745, returning to step 814 for continued execution.When the thread state is no longer valid (i.e., the thread is paused) instep 828, with the paused status for that thread ID indicated in thethread memory 720, execution of that thread is suspended, step 830,until the status for that thread ID returns to valid, step 832, and thethread ID (with its valid state and priority) is returned to theexecution (ready-to-run) queue 745, returning to step 814 for continuedexecution.

Similarly, the HTP 300 may generate calls, such as to create threads onlocal or remote compute elements, such as to create threads on otherHTPs 300 or HTF circuits 200. Such calls are also created as outgoingdata packets, and more specifically as outgoing work descriptor packetson the first interconnection network 150. For example, an instruction ofa current thread being executed may be a “fiber create” instruction(stored as a possible instruction in the instruction cache 740), tospawn a plurality of threads for execution on the various computeresources. As discussed in greater detail below, such a fiber createinstruction designates (using an address or virtual address (nodeidentifier)) what computing resource(s) will execute the threads, andwill also provide associated arguments. When the fiber createinstruction is executed in the execution pipeline 750, the fiber createcommand is pushed into the network command queue 790, and the nextinstruction is executed in the execution pipeline 750. The command ispulled out of the network command queue 790, and the data packet encodercircuit 780 has the information needed to create and send a workdescriptor packet to the specified destination HTF 200 or HTP 300.

If the created threads will have return arguments, then such aninstruction will also allocate and reserve associated memory space, suchas in the return argument buffers 734. If there is insufficient space inthe return argument buffers 734, the instruction will be paused untilreturn argument buffers 734 are available. The number of fibers orthreads created is only limited by the amount of space to hold theresponse arguments. Created threads that do not have return argumentscan avoid reserving return argument space, avoiding the possible pausestate. This mechanism ensures that returns from completed threads alwayshave a place to store their arguments. As the returns come back to theHTP 300 as data packets on the first interconnection network 150, thosepackets are decoded, as discussed above, with the return data stored inthe associated, reserved space in the return argument buffers 734 of thethread memory 720, as indexed by the thread ID associated with the fibercreate instruction. As many registers could be utilized for the returnargument, the return argument buffers 734 can be provided as a link listof all the spawned threads or return argument buffers or registersallocated for that thread ID. Significantly, this mechanism can allowpotentially thousands of threads to be created very quickly, effectivelyminimizing the time involved in a transition from a single threadexecution to high thread count parallelism.

As discussed in greater detail below, various types of fiber joininstructions are utilized to determine when all of the spawned threadshave completed, and can be an instruction with or without waiting. Acount of the number of spawned threads is maintained in the pendingfiber return count registers 732, which count is decremented as threadreturns are received by the HTP 300. A join operation can be carried outby copying the returns into the registers associated with the spawningthread ID. If the join instruction is a waiting instruction, it willstay in a paused state until the return arrives which designates thatthread ID of the spawning thread. In the interim, other instructions areexecuted by the execution pipeline 750 until the pause state of the joininstruction changes to a valid state and the join instruction isreturned to the execution (ready-to-run) queue 745.

A thread return instruction may also be utilized as the instructionfollowing the fiber create instruction, instead of a join instruction.When the count in the pending fiber return count registers 732 reacheszero, with the receipt of the last thread return data packet, a threadreturn instruction may also be executed, and indicates that the fibercreate operation has been completed and all returns received, allowingthe thread ID, the return argument buffers 734, and link list to befreed for other uses. In addition, it may also generate and transmit awork descriptor return packet (e.g., having result data) to the sourcewhich called the main thread (e.g., to the identifier or address of thesource which generated the call).

The join all instruction does not require that arguments be returned,only acknowledgements which decrement the count in the pending fiberreturn count registers 732. When that count reaches zero, that thread isrestarted, as the join all is now complete.

Communication between processing elements is required to facilitateprocessing of parallel algorithms. The representative embodimentsprovide an efficient means for threads of a set of processing resourcesto communicate, using various event messages, which may also includedata (such as arguments or results). The event messaging allows any hostprocessors 110 with hardware maintained cache coherency and anyacceleration processors (such as the HTP 300) with software maintainedcache coherency to efficiently participate in event messaging.

The event messaging supports both point to point and broadcast eventmessages. Each processing resource (HTP 300) can determine when areceived event operation has completed and the processing resourceshould be informed. The event receive modes include simple (a singlereceived event completes the operation), collective (a counter is usedto determine when sufficient events have been received to complete theoperation), and broadcast (an event received on a specific channelcompletes the event). Additionally, events can be sent with an optional64-bit data value.

The HTP 300 has a set of event receive states, stored in the event stateregisters 744, that consists of a 2-bit receive mode, a 16-bitcounter/channel number, and a 64-bit event data value. AN HTP 300 canhave multiple sets of event receive states per thread context, whereeach set is indexed by an event number. Thus, an event can be targetedto a specific thread (thread ID) and event number. The sent event can bea point-to-point message with a single destination thread, or abroadcast message sent to all threads within a group of processingresources belonging to the same process. When such events are received,the paused or sleeping thread can be reactivated to resume processing.

This use of event state registers 744 is much more efficient than astandard Linux based host processor, which can send and receive eventsthrough an interface that allows the host processor 110 to periodicallypoll on completed receive events. Threads waiting on event messages canpause execution until the receive operation completes, i.e., the HTP 300can pause execution of threads pending the completion of receive events,rather than waste resources by polling, allowing other threads to beexecuting during these intervals. Each HTP 300 also maintains a list ofprocessing resources that should participate in receiving events toavoid process security issues.

A point-to-point message will specify an event number and thedestination (e.g., node number, which HTP 300, which core, and whichthread ID). On the receive side, an HTP 300 will have been configured orprogrammed with one or more event numbers held in the event stateregisters 744. If that HTP 300 receives an event message having thatevent number, it is triggered and transitions from a paused state to avalid state to resume execution, such as executing an event receivedinstruction (e.g., EER, below). That instruction will then determine ifthe correct event number was received, and if so, write any associated64-bit data into general-purpose registers 728, for use by anotherinstruction. If the event received instruction executes and the correctevent number was not received, it will be paused until that specificevent number is received.

An event listen (EEL) instruction may also be utilized, with an eventmask stored in the event received mask registers 742, indicating one ormore events which will be used to trigger or wake up the thread. When anevent message with any of those designated events arrives, the receivingHTP 300 will know which event number was triggered, e.g., what otherprocess may have been completed, and will receive event data from thosecompleted events. The event listen instruction may also have waiting anda no waiting variations, as discussed in greater detail below.

For event messaging in a collective mode, the receiving HTP 300 willcollect (wait for) a set of receive events before triggering, setting acount in the event state registers 744 to the value required, which isdecremented as the required event messages are received, and triggeringonce the count has been decremented to zero.

In a broadcast mode, a sender processing resource can transmit a messageto any thread within the node. For example, a sending HTP 300 maytransmit a series of point-to-point messages to each other HTP 300within the node, and each receiving HTP 300 will then pass the messageto each internal core 705. Each core control circuit 710 will go throughits thread list to determine if it corresponds to an event number whichit has been initialized to receive, and upon which channel that may havebeen designated on the first interconnection network 150.

This broadcast mode is especially useful when thousands of threads maybe executing in parallel, in which the last thread to execute transmitsa broadcast event message indicating completion. For example, a firstcount of all threads requiring completion may be maintained in the eventstate registers 744, while a second count of all threads which haveexecuted may be maintained in memory 125. As each thread executes, italso performs a fetch and increment atomic operation on the secondcount, such as through an atomic operation of the memory 125 (andcompares it to the first count), and sets its mode to receive abroadcast message by executing an EER instruction to wait until itreceives a broadcast message. The last one to execute will see thefetched value of the second count as the required first count minus one,indicating that it is the last thread to execute, and therefore sendsthe broadcast message, which is a very fast and efficient way toindicate completion of significant parallel processing.

As mentioned above, while the HTP 300 may utilize standard RISC-Vinstructions, a significantly extended set of instructions are providedto take advantage of all the system 100 compute resources, as discussedin greater detail below. Threads created from the host processor 110 aretypically referred to as master threads, and threads created from theHTP 300 are typically referred to as fibers or fiber threads, and allare executed identically on the destination HTP 300 and HTF 200, withoutgoing through the memory 125.

New Load Instructions:

The HTP 300 has a comparatively small number of read/write buffers perthread, also referred to as data cache registers 746. The buffers (datacache registers 746) temporarily store shared memory data for use by theowning thread. The data cache registers 746 are managed by a combinationof hardware and software. Hardware automatically allocates buffers andevicts data when needed. Software, through the use of RISC-Vinstructions decides which data should be cached (read and write data),and when the data cache registers 746 should be invalidated (if clean)or written back to memory (if dirty). The RISC-V instruction setprovides a FENCE instruction as well as acquire and release indicatorson atomic instructions.

The standard RISC-V load instructions automatically use the read datacache registers 746. A standard load checks to see if the needed data isin an existing data cache register 746. If it is then the data isobtained from the data cache register 746 and the executing thread isable to continue execution without pausing. If the needed data is not ina data cache register 746, then the HTP 300 finds an available datacache register 746 (evicting data from a buffer needed), and reads64-bytes from memory into the data cache register 746. The executingthread is paused until the memory read has completed and the load datais written into a RISC-V register.

Read buffering has two primary benefits: 1) larger accesses are moreefficient for the memory controller 120, and 2) accesses to the bufferallow the executing thread to avoid stalling. However, there aresituations when using the buffer causes problems. An example is a gatheroperation where accesses would typically cause thrashing of the datacache registers 746. For this reason, a set of special load instructionsare provided to force a load instruction to check for a cache hit, buton a cache miss to issue a memory request for just the requested operandand not put the obtained data in a data cache register 746, and insteadput the data into one of the general-purpose registers 728.

The new load instruction provides for “probabilistic” caching based uponanticipated frequency of access, for frequently used data versussparsely or rarely used data. This is especially significant for usewith sparse data sets, which if put into the data cache registers 746,would overwrite other data which will be needed again more frequently,effectively polluting the data cache registers 746. The new loadinstruction (NB or NC) allows frequently used data to remain in the datacache registers 746, and less frequently used (sparse) data which wouldbe typically cached to be designated instead for non-cached storage inthe general-purpose registers 728.

Instructions of this type have an NB suffix (non-buffered) (orequivalently, an NC suffice (non-cached):

LB.NB RA,40(SP).

The NB (NC) load instructions are expected to be used in runtimelibraries written in assembly.

The following load instructions were added as 32 bit instructions, whereImm is the immediate field, RA is a register name, rs1 is a sourceindex, rd is a destination index, and the bits in fields 14-12 and 6-0specify the instruction, in Table 8.

TABLE 8 31 20 19 15 14 12 11 7 6 0 Imm[11:0] rs1 000 rd 0000010 LBNBImm[11:0] rs1 001 rd 0000010 LH.NB Imm[11:0] rs1 010 rd 0000010 LW.NBImm[11:0] rs1 011 rd 0000010 LD.NB Imm[11:0] rs1 100 rd 0000010 LBU.NBImm[11:0] rs1 101 rd 0000010 LHU.NB Imm[11:0] rs1 110 rd 0000110 LWU.NBImm[11:0] rs1 010 rd 0000110 FLW.NB Imm[11:0] rs1 011 rd 0000110 FLD.NB

Bandwidth to memory is often the major contributor to limiting anapplication's performance. The representative embodiments provides ameans to inform the HTP 300 as to how large of a memory load requestshould be issued to memory 125. The representative embodiments reducewasted memory and bandwidth of the first interconnection network 150 dueto access memory data that is not used by the application.

A further optimization exists where an application knows the size of adata structure being accessed and can specify the amount of data to beloaded into a data cache register 746. As an example, if an algorithmuses a structure that is 16-bytes in size, and the structures arescattered in memory, then it would be optimal to issue 16-byte memoryreads and place the data into a data cache register 746. Therepresentative embodiments define a set of memory load instructions thatprovide both the size of the operand to be loaded into an HTP 300register, and the size of the access to memory if the load misses thedata cache register 746. The actual load to memory 125 may be smallerthan the instruction specified size if the memory access would cross acache line boundary. In this case, the access size is reduced to ensurethat the response data is written to a single cache line of the datacache registers 746.

When the requested data would be less than a cache line, the loadinstruction may also request additional data that the HTP 300 iscurrently unneeded but likely to be needed in the future, which is worthobtaining at the same time (e.g., as a pre-fetch), optimizing the readsize access to memory 125. This instruction can also override anyreductions in access size which might have been utilized (as discussedin greater detail below with reference to FIG. 32 ) for bandwidthmanagement.

The representative embodiments therefore minimize wasted bandwidth byonly requesting memory data that is known to be needed. The result is anincrease in application performance.

A set of load instructions have been defined that allow the amount ofdata to be accessed to be specified. The data is written into a buffer,and invalidated by an eviction, a FENCE, or an atomic with acquirespecified. The load instructions provide hints as to how much additionaldata (in 8-byte increments) is to be accessed from memory and written tothe memory buffer. The load will only access additional data to the next64-byte boundary. A load instruction specifies the number of additional8-byte elements to load using the operation suffix RB0-RB7:

LD.RB7 RA,40(SP)

The instruction formats are shown in Table 9. The number of 8-byte dataelements to load into the buffer is specified as bits 6 and 4:3 of the32-bit instruction. These load instructions can be used in assemblywritten routines, or ideally by a complier. It is expected thatinitially only hand written assembly will take advantage of theseinstructions.

TABLE 9 31 20 19 15 14 12 11 7 6 0 Imm[11:0] rs1 000 rd x0xx010 LB.RC1-7Imm[11:0] rs1 001 rd x0xx010 LH.RC1-7 Imm[11:0] rs1 010 rd x0xx010LW.RC1-7 Imm[11:0] rs1 011 rd x0xx010 LD.RC1-7 Imm[11:0] rs1 100 rdx0xx010 LBU.RC1-7 Imm[11:0] rs1 101 rd x0xx010 LHU.RC1-7 Imm[11:0] rs1110 rd x0xx010 LWU.RC1-7 Imm[11:0] rs1 010 rd x0xx110 FLW.RC1-7Imm[11:0] rs1 011 rd x0xx110 FLD.RC1-7

New Store Instructions

The HTP 300 has a small number of memory buffers that temporarily storeshared memory data. The memory buffers allow multiple writes to memoryto be consolidated into a smaller number of memory write requests. Thishas two benefits: 1) the fewer write requests is more efficient for thefirst interconnection network 150 and memory controllers 120, and 2) anHTP 300 suspends the thread that performs a memory store until the datais stored to either the HTP 300 memory buffer, or at the memorycontroller 120. Stores to the HTP 300 memory buffer are very quick andwill typically not cause the thread to suspend execution. When a bufferis written to the memory controller 120, then the thread is suspendeduntil a completion is received in order to ensure memory 125consistency.

The standard RISC-V store instructions write data to the HTP 300 memorybuffers. However, there are situations in which it is known that it isbetter to write the data directly to memory and not write to a memorybuffer. One such situation is a scatter operation. A scatter operationwould typically write just a single data value to the memory buffer.Writing to the buffer causes the buffers to thrash and other store datathat would benefit from write coalescing is forced back to memory. A setof store instructions are defined for the HTP 300 to indicate that writebuffering should not be used. These instructions write data directly tomemory 125, causing the executing thread to be paused until the writecompletes.

The store no buffering instructions are expected to be used in handassembled libraries and are indicated with a NB suffix:

ST.NB RA,40(SP)

The following store instructions were added as shown in Table 10.

TABLE 10 31 25 24 20 19 15 14 12 11 7 6 0 Imm[11:5] rs2 rs1 000 Imm[4:0]0100010 SB.NB Imm[11:5] rs2 rs1 001 Imm[4:0] 0100010 SH.NB Imm[11:5] rs2rs1 010 Imm[4:0] 0100010 SW.NB Imm[11:5] rs2 rs1 011 Imm[4:0] 0100010SD.NB Imm[11:5] rs2 rs1 010 Imm[4:0] 0100110 FSW.NB Imm[11:5] rs2 rs1011 Imm[4:0] 0100110 FSD.NB

Custom Atomic Store and Clear Lock (CL) Instructions:

Custom atomic operations set a lock on the provided address when theatomic operation is observed by the memory controller. The atomicoperation is performed on an associated HTP 300. The HTP 300 shouldinform the memory controller when the lock should be cleared. Thisshould be on the last store operation that the HTP 300 performs for thecustom atomic operation (or on a fiber terminate instruction if no storeis required). The HTP 300 indicates that the lock is to be cleared byexecuting a special store operation. The store and clear lockinstructions.

The following sequence of instructions could be used to implement acustom atomic DCAS operation:

// a0 - atomic address // a1 - 64-bit memory value of a0 // a2 - DCAScompare value 1 // a3 - DCAS compare value 2 // a4 - DCAS swap value 1// a5 - DCAS swap value 2 atomic_dcas: bne a1, a2, fail  // first 8-bytecompare ld.nb a6, 8(a0)  // load second 8-byte memory value - should hitmemory cache bne a6, a3, fail  // second 8-byte compare sd a4, 0(a0)  //store first 8-byte swap value to thread store buffer sd.cl a5, 8(a0)  //store second 8-byte value and clear memory lock eft x0  // AMO successresponse fail: li a1, 1 eft.cl a1,(a0)  // AMO failure response (andclear memory lock)The store instructions that indicate the lock should be cleared are:

SB.CL RA,40(SP)

SH.CL RA,40(SP)

SW.CL RA,40(SP)

SD.CL RA,40(SP)

FSW.CL RA,40(SP)

FSD.CL RA,40(SP)

The format for these store instructions is shown Table 11.

TABLE 11 31 25 24 20 19 15 14 12 11 7 6 0 Imm[11:5] rs2 rs1 000 Imm[4:0]0110010 SB.CL Imm[11:5] rs2 rs1 001 Imm[4:0] 0110010 SH.CL Imm[11:5] rs2rs1 010 Imm[4:0] 0110010 SW.CL Imm[11:5] rs2 rs1 011 Imm[4:0] 0110010SD.CL Imm[11:5] rs2 rs1 010 Imm[4:0] 0110110 FSW.CL Imm[11:5] rs2 rs1011 Imm[4:0] 0110110 FSD.CL

atomic_float_add:  fadd.d a2, a1, a2  // a1 contains memory value, a2contains value to be added in  fsd.cl a2, 0(a0)  // a0 contains memoryaddress, clear lock and terminate atomic  eft  // evict all line frombuffer, terminate atomic thread

Fiber Create Instructions:

The Fiber Create (“EFC”) instruction initiates a thread on an HTP 300 orHTF 200.

EFC.HTP.A4

EFC.HTF.A4

This instruction performs a call on an HTP 300 (or HTF 200), beginsexecution at the address in register a0. (Optionally, a suffix DA may beutilized. The instruction suffix DA indicates that the target HTP 300 isdetermined by the virtual address in register a1. If the DA suffix isnot present, then an HTP 300 on the local system 100 is targeted.) Thesuffix A1, A1, A2 and A4 specifies the number of additional arguments tobe passed to the HTP 300 or HTF 200. The argument count is limited tothe values 0, 1, 2, or 4 (e.g., a packet should fit in 64B). Theadditional arguments are from register state (a2-a5).

It should be noted that if a return buffer is not available at the timethe EFC instruction is executed, then the EFC instruction will waituntil a return argument buffer is available to begin execution. Once theEFC instruction successfully creates a fiber, the thread continues atthe instruction immediately following the EFC instruction.

It also should be noted that threads created by the host processor 110are allowed to execute the EFC instruction and create fibers. Fiberscreated by an EFC instruction are not allowed to execute the EFCinstruction and will force an exception, optionally. The format forthese fiber create instructions is shown Table 12.

TABLE 12 31 25 24 20 19 15 14 12 11 7 6 0 0000000 00000 00000 ac 000001110010 EFC.HTP 0001000 00000 00000 ac 00000 1110010 EFC.HTP.DA 001000000000 00000 ac 00000 1110010 EFC.HTF 0011000 00000 00000 ac 000001110010 EFC.HTF.DA ac Encoding Suffix Argument Count 0 No suffix 0 1 .A11 2 .A2 2 3 .A4 4

Thread Return Instructions:

The Thread Return (ETR) instruction passes arguments back to the parentthread that initiated the current thread (through a host processor 110thread create or HTP 300 fiber create). Once the thread has completedthe return instruction, the thread is terminated.

ETR.A2

This instruction performs a return to an HTP 300 or host processor 110.The ac suffix specifies the number of additional arguments to be passedto the HTP or host. Argument count can be the values 0, 1, 2 or 4. Thearguments are from register state (a0-a3). The format for these threadreturn instructions is shown Table 13.

TABLE 13 31 25 24 20 19 15 14 12 11 7 6 0 0100000 00000 00000 ac 000001110010 EFR ac Encoding Suffix Argument Count 0 No suffix 0 1 .A1 1 2.A2 2 3 .A4 4

Fiber Join Instructions:

The Fiber Join (EFJ) instruction checks to see if a created fiber hasreturned. The instruction has two variants, join wait and non-wait. Thewait variant will pause thread execution until a fiber has returned. Thejoin non-wait does not pause thread execution but rather provides asuccess/failure status. For both variants, if the instruction isexecuted with no outstanding fiber returns then an exception isgenerated.

The arguments from the returning fiber (up to four) are written toregisters a0-a3.

EFJ

EFJ.NW

The format for these fiber join instructions is shown Table 14.

TABLE 14 31 25 24 20 19 15 14 12 11 7 6 0 0101000 00000 00000 000 000001110010 EFJ 0110000 00000 00000 000 00000 1110010 EFJ.NW

Fiber Join All Instructions:

The Fiber Join All instruction (EFJ.ALL) pends until all outstandingfibers have returned. The instruction can be called with zero or morepending fiber returns. No instruction status or exceptions aregenerated. Any returning arguments from the fiber returns are ignored.

EFJ.ALL

The format for these fiber join all instructions is shown Table 15.

TABLE 15 31 25 24 20 19 15 14 12 11 7 6 0 0111000 00000 00000 000 000001110010 EFJ.ALL

Atomic Return Instructions:

The system 100 atomic return instruction (EAR) is used to complete theexecuting thread of a custom atomic operation and possibly provide aresponse back to the source that issued the custom atomic request.

The EAR instruction can send zero, one, or two 8-byte arguments valueback to the issuing compute element. The number of arguments to sendback is determine by the ac2 suffix (A1 or A2). No suffix means zeroarguments, A1 implies a single 8-byte argument, and A2 implies two8-byte arguments. The arguments, if needed, are obtained from Xregisters a1 and a2.

The EAR instruction is also able to clear the memory line lockassociated with the atomic instruction. The EAR uses the value in the a0register as the address to send the clear lock operation. The clear lockoperation is issued if the instruction contains the suffix CL.

The following DCAS example sends a success or failure back to therequesting processor using the EAR instruction:

// a0 - atomic address // a1 - 64-bit memory value of a0 // a2 - DCAScompare value 1 // a3 - DCAS compare value 2 // a4 - DCAS swap value 1// a5 - DCAS swap value 2 atomic_dcas: bne a1, a2, fail // first 8-bytecompare ld.nb a6, 8(a0) // load second 8-byte memory value - should hitmemory cache bne a6, a3, fail // second 8-byte compare sd a4, 0(a0) //store first 8-byte swap value to thread store buffer sd.cl a5, 8(a0) //store second 8-byte value and clear memory lock li a1, 0 ear.a1 // AMOsuccess response fail: li a1, 1 ear.cl.a1 // AMO failure response (andclear memory lock)

The instruction has two variants that allow the EFT instruction to alsoclear the memory lock associated with the atomic operation. The formatfor the supported instructions is shown in Table 16.

TABLE 16 31 25 24 20 19 15 14 12 11 7 6 0 1010000 00000 00000 ac2 000001110010 EAR 1011000 00000 00000 ac2 00000 1110010 EAR.CL ac2 EncodingSuffix Argument Count 0 No suffix 0 1 .A1 1 2 .A2 2

First and Second Priority Instructions:

The second (or low) priority instruction transitions the current threadhaving a first priority to a second, low priority. The instruction isgenerally used when a thread is polling on an event to occur (i.e.barrier).

ELP

The format for the ELP instruction is shown Table 17.

TABLE 17 31 25 24 20 19 15 14 12 11 7 6 0 1000000 00000 00000 000 000001110010 ELP

The first (or high) priority instruction transitions the current threadhaving a second (or low) priority to a first (or high or normal)priority. The instruction is generally used when a thread is polling andan event has occurred (i.e. barrier).

ENP

The format for the ENP instruction is shown Table 18.

TABLE 18 31 25 24 20 19 15 14 12 11 7 6 0 1001000 00000 00000 000 000001110010 ENP

Floating Point Atomic Memory Operations:

Floating point atomic memory operations are performed by the HTP 300associated with a memory controller 120. The floating point operationsperformed are MIN, MAX and ADD, for both 32 and 64-bit data types.

The aq and rl bits in the instruction specify whether all write data isto be visible to other threads prior to issuing the atomic operation(aq), and whether all previously written data should be visible to thisthread after the atomic completes (rl). Put another way, the aq bitforces all write buffers to be written back to memory, and the rl bitforces all read buffers to be invalidated. It should be noted that rs1is an X register value, whereas rd and rs2 are F register values.

AMOFADD.S rd, rs2, (rs1)

AMOFMIN.S rd, rs2, (rs1)

AMOFMAX.S rd, rs2, (rs1)

AMOFADD.D rd, rs2, (rs1)

AMOFMIN.D rd, rs2, (rs1)

AMOFMAX.D rd, rs2, (rs1)

The format for these floating point atomic memory operation instructionsis shown Table 19.

TABLE 19 31 27 26 25 24 20 19 15 14 12 11 7 6 0 00000 aq rl rs2 rs1 010rd 0101110 AMOFADD.S 00001 aq rl rs2 rs1 010 rd 0101110 AMOFMIN.S 00010aq rl rs2 rs1 010 rd 0101110 AMOFMAX.S 00000 aq rl rs2 rs1 011 rd0101110 AMOFADD.D 00001 aq rl rs2 rs1 011 rd 0101110 AMOFMIN.D 00010 aqrl rs2 rs1 011 rd 0101110 AMOFMAX.D

Custom Atomic Memory Operations:

Custom atomic operations are performed by the HTP 300 associated with amemory controller 120. The operation is performed by executing RISC-Vinstructions. Up to 32 custom atomic operations can be available withinthe memory controllers 120 of a system 100. The custom atomics are asystem wide resource, available to any process attached to the system100.

The aq and rl bits in the instruction specify whether all write data isto be visible to other threads prior to issuing the atomic operation(rl), and whether all previously written data should be visible to thisthread after the atomic completes (aq). Put another way, the rl bitforces all write buffers to be written back to memory, and the aq bitforces all read buffers to be invalidated.

The custom atomics use the a0 register to specify the memory address.The number of source arguments is provided by the suffix (A0, A1, A2 orA4), and are obtained from registers a1-a4. The number of result valuesreturned from memory can be 0-2, and is defined by the custom memoryoperation. The result values are written to register a0-a1.

AMOCUST0.A4

The following custom atomic instructions are defined as shown in Table20.

TABLE 20 31 27 26 25 24 20 19 15 14 12 11 7 6 0 10000 aq rl 00000 00000ac 00000 0101110 AMOCUST0 10001 aq rl 00000 00000 ac 00000 0101110AMOCUST1 10010 aq rl 00000 00000 ac 00000 0101110 AMOCUST2 10011 aq rl00000 00000 ac 00000 0101110 AMOCUST3 10100 aq rl 00000 00000 ac 000000101110 AMOCUST4 10101 aq rl 00000 00000 ac 00000 0101110 AMOCUST5 10110aq rl 00000 00000 ac 00000 0101110 AMOCUST6 10111 aq rl 00000 00000 ac00000 0101110 AMOCUST7

The ac field is used to specify the number of arguments (0, 1, 2, or 4).The following Table 21 shows the encodings.

TABLE 21 ac Argument Encoding Suffix Count 0 No suffix 0 1 .A1 1 2 .A2 23 .A4 4There are eight custom atomic instructions defined, with 4 argumentcount variants each, resulting a total of 32 possible custom atomicoperators.

Event Management:

The system 100 is an event driven architecture. Each thread has a set ofevents that is able to monitor, utilizing the event received maskregisters 742 and the event state registers 744. Event 0 is reserved fora return from a created fiber (HTP 300 or HTF 200). The remainder of theevents are available for event signaling, either thread-to-thread,broadcast, or collection. Thread-to-thread allows a thread to send anevent to one specific destination thread on the same or a differentnode. Broadcast allows a thread to send a named event to a subset ofthreads on its node. The receiving thread should specify which namedbroadcast event it is expecting. Collection refers to the ability tospecify the number of events that are to be received prior to the eventbecoming active.

An event triggered bit can be cleared (using the EEC instruction), andall events can be listened for (using the EEL instruction). The listenoperation can either pause the thread until an event has triggered, orin non-waiting mode (NW) allowing a thread to periodically poll whileother execution proceeds.

A thread is able to send an event to a specific thread using the eventsend instruction (EES), or broadcast an event to all threads within anode using the event broadcast instruction (EEB). Broadcasted events arenamed events where the sending thread specifies the event name (a 16-bitidentifier), and the receiving threads filter received broadcast eventsfor a pre-specified event identifier. Once received, the event should beexplicitly cleared (EEC) to avoid receiving the same event again. Itshould be noted that all event triggered bits are clear when a threadstarts execution.

Event Mode Instructions:

The event mode (EEM) instruction sets the operation mode for an event.Event 0 is reserved for thread return events, the remainder of theevents can be in one of three receive modes: simple, broadcast, orcollection.

In simple mode, a received event immediately causes the triggered bit tobe set and increments the received message count by one. Each newlyreceived event causes the received event count to be incremented. Thereceive event instruction (EER) causes the received event count to bedecremented by one. The event triggered bit is cleared when the counttransitions back to zero.

In broadcast mode, a received event's channel is compared to the eventnumber's broadcast channel. If the channels match, then the eventtriggered bit is set. The EER instruction causes the triggered bit to becleared.

In collection mode, received event causes the event trigger count to bedecremented by one. When the count reaches zero, then the eventtriggered bit is set. The EER instruction causes the triggered bit to becleared.

The EEM instruction prepares the event number for the chosen mode ofoperation. In simple mode, the 16-bit event counter is set to zero. Forbroadcast mode, the 16-bit event channel number is set to the valuespecified by the EEM instruction. For collection mode, the 16-bit eventcounter is set to the value specified by the EEM instruction. Each ofthe three modes use the same 16-bit value differently.

EEM.BM rs1, rs2 ;rs1 = event #, rs2 = broadcast channel EEM.CM rs1, rs2;rs1 = event #, rs2 = collection count EEM.SM rs1 ;rs1 = event #

The format for the event mode instruction is shown Table 22.

TABLE 22 31 25 24 20 19 15 14 12 11 7 6 0 0000100 rs2 rs1 000 000001110010 EEM.BM 0001100 rs2 rs1 000 00000 1110010 EEM.CM 0010100 00000rs1 000 00000 1110010 EEM.SM

Event Destination Instruction:

The event destination (EED) instruction provides an identifier for anevent within the executing thread. The identifier is unique across allexecuting threads within a node. The identifier can be used with theevent send instruction to send an event to the thread using the EESinstruction. The identifier is an opaque value that contains theinformation needed to send the event from a source thread to a specificdestination thread.

The identifier can also be used to obtain a unique value for sending abroadcast event. The identifier includes space for an event number. Theinput register rs1 specifies the event number to encode within thedestination thread identifier. The output rd register contains theidentifier after the instruction executes.

EED rd, rs1

The format for the event destination instruction is shown Table 23.

TABLE 23 31 25 24 20 19 15 14 12 11 7 6 0 0011100 00000 rs1 000 rd1110010 EED

The event destination instruction can also be utilized by a process toobtain its own address, which can then be used in other broadcastmessages, for example, to enable that process to receive other eventmessages as a destination, e.g., for receiving return messages when theprocess is a master thread.

Event Send Instructions:

The event send (EES) instruction sends an event to a specific thread.Register rs1 provides the destination thread and event number. Registerrs2 provides the optional 8-byte event data.

EES rs1

EES.A1 rs1, rs2 The rs2 register provides the target HTP 300 for theevent send operation. Register rs1 provides the event number to be sent.Legal values for rs1 are 2-7. The format for the event send instructionis shown Table 24.

TABLE 24 31 25 24 20 19 15 14 12 11 7 6 0 0100100 00000 rs1 000 000001110010 EES 0101100 rs2 rs1 000 00000 1110010 EES.A1

Event Broadcast Instructions:

The event broadcast (EEB) instruction broadcasts an event to all threadswithin the node. Register rs1 provides the broadcast channel to be sent(0-65535). Register rs2 provides optional 8-byte event data.

EEB rs1

EEB.A1 rs1, rs2

The format for the event broadcast instruction is shown Table 25.

TABLE 25 31 25 24 20 19 15 14 12 11 7 6 0 0110100 00000 rs1 000 000001110010 EEB 0111100 rs2 rs1 000 00000 1110010 EEB.A1

Event Listen Instructions:

The event listen (EEL) instruction allows a thread to monitor the statusof received events. The instruction can operate in one of two modes:waiting and non-waiting. The waiting mode will pause the thread until anevent is received, the non-waiting mode provides the received events atthe time the instruction is executed.

EEL rd, rs1

EEL.NW rd, rs1

Register rs1 provides a mask of available events as the output of thelisten operation. The non-waiting mode will return a value of zero inrs1 if no events are available. The format for the event listeninstructions is shown Table 26.

TABLE 26 31 25 24 20 19 15 14 12 11 7 6 0 1000100 00000 rs1 000 rd1110010 EEL 1001100 00000 rs1 000 rd 1110010 EEL.NW

Event Receive Instructions:

The event receive (EER) instruction is used to receive an event.Receiving an event includes acknowledging that an event was observed,and receiving the optional 8-byte event data. Register rs1 provides theevent number. Register rd contains optional 8-byte event data.

EER rs1

EER.A1 rd, rs1

The format for the event receive instructions is shown Table 27.

TABLE 27 31 25 24 20 19 15 14 12 11 7 6 0 1010100 00000 rs1 000 000001110010 EER 1011100 00000 rs1 000 rd 1110010 EER.A1

HTP 300 instruction formats are also provided for call, fork or transferinstructions, previously discussed.

Send Call Instruction:

The Thread Send Call instruction initiates a thread on an HTP 300 or HTF200 and pauses the current thread until the remote thread performs areturn operation:

HTSENDCALL.HTP.DA Ra, Rb, Args.

The Thread Send Call instruction performs a call on an HTP 300, beginsexecution at the address in register Ra. The instruction suffix DAindicates that the target HTP 300 is determined by the virtual addressin register Rb. If the DA suffix is not present, then an HTP 300 on thelocal node is targeted. The constant integer value Args identifies thenumber of additional arguments to be passed to the remote HTP 300. Argsis limited to the values 0 through 4 (e.g., a packet should fit in 64B).The additional arguments are from register state. It should be notedthat if a return buffer is not available at the time the HTSENDCALLinstruction is executed, then the HTSENDCALL instruction will wait untila buffer is available to begin execution. Once the HTSENDCALL hascompleted, the thread is paused until a return is received. When thereturn is received, the thread is resumed at the instruction immediatelyfollowing the HTSENDCALL instruction. The instruction sends a firstinterconnection network 150 packet containing the following values,shown in Table 28:

TABLE 28 FIELD SIZE DESCRIPTION PROCESS ID 32 b Process ID of processRETURN NODE ID 16 b Return packet destination node RETURN COMPONENT ID10 b Return packet fabric endpoint RETURN CONTEXT ID  8 b Context ID ofsource HTP thread INSTRUCTION ADDRESS  64 b Target HTP instructionaddress OPTIONAL ARGUMENTS 1-4 0-256 b Optional Arguments 1-4

Thread Fork Instruction:

The Thread Fork instruction initiates a thread on an HTP 300 or HTF 200and continues the current thread:

HTSENDFORK.HTF.DA Ra, Rb, Args.

The Thread Fork instruction performs a call on an HTF 200 (or HTP 300),begins execution at the address in register Ra. The instruction suffixDA indicates that the target HTF 200 is determined by the Node ID withinthe virtual address in register Rb. If the DA suffix is not present,then an HTF 200 on the local node is targeted. The constant integervalue Args identifies the number of additional arguments to be passed tothe remote HTF. Args is limited to the values 0 through 4 (e.g., apacket should fit in 64B). The additional arguments are from registerstate. It should be noted that if a return buffer is not available atthe time the HTSENDFORK instruction is executed, then the HTSENDFORKinstruction will wait until a buffer is available to begin execution.Once the HTSENDFORK has completed, the thread continues execution at theinstruction immediately following the HTSENDFORK instruction. The ThreadFork instruction sends a first interconnection network 150 packetcontaining the following values, shown in Table 29:

TABLE 29 FIELD SIZE DESCRIPTION PROCESS ID 32 b Process ID of processRETURN NODE ID 16 b Return packet destination node RETURN COMPONENT ID10 b Return packet fabric endpoint RETURN CONTEXT ID  8 b Context ID ofsource HTP 300 thread KERNEL ADDRESS 60 b Target HTF 200 kernel address(bits 3-0 are implicitly zero) ENTRY INSTRUCTION  4 b HTF entryinstruction OPTIONAL ARGUMENTS 1-4 0-256 b Optional Arguments 1-4

Thread Transfer Instruction:

The Thread Transfer instruction initiates a thread on an HTP 300 or HTF200 and terminates the current thread:

HTSENDXFER.HTP.DA Ra, Rb, Args.

The Thread Transfer instruction performs a transfer to an HTP 300 andbegins execution at the address in register Ra. The instruction suffixDA indicates that the target HTP 300 is determined by the virtualaddress in register Rb. If the DA suffix is not present, then an HTP 300on the local node is targeted. The constant integer value Argsidentifies the number of additional arguments to be passed to the remoteHTP 300. Args is limited to the values 0 through 4 (packet must fit in64B). The additional arguments are from register state. Once theHTSENDXFER has completed, the thread is terminated. The Thread Transferinstruction sends a first interconnection network 150 packet containingthe following values shown in Table 30:

TABLE 30 FIELD SIZE DESCRIPTION PROCESS ID 32 b Process ID of processRETURN NODE ID 16 b Return packet destination node RETURN COMPONENT ID10 b Return packet fabric endpoint RETURN CONTEXT ID  8 b Context ID ofsource HTP thread INSTRUCTION ADDRESS 64 b Target HTP instructionaddress OPTIONAL ARGUMENTS 1-4 0-256 b Optional Arguments 1-4

Receive Return Instruction:

The thread receive return instruction HTRECVRTN.WT checks to see if areturn for the thread has been received. If the WT suffix is present,then the receive return instruction will wait until a return has beenreceived. Otherwise a testable condition code is set to indicate thestatus of the instruction. When a return is received, the return'sarguments are loaded into registers. The instruction immediatelyfollowing the HTRECVRTN instruction is executed after the returninstruction completes.

FIG. 30 is a detailed block diagram of a representative embodiment of athread selection control circuitry 805 of the control logic and threadselection circuitry 730 of the HTP 300. As mentioned above, a second orlow priority queue 760 is provided, and thread IDs are selected from thefirst (or high) priority queue 755 or the second or low priority queue760 using a thread selection multiplexer 785, under the control of thethread selection control circuitry 805. Threads in the second priorityqueue 760 are pulled from the queue and executed at a lower rate thanthreads in the first priority queue 760.

As mentioned above, a pair of instructions, ENP and ELP, are used totransition a thread from a first priority to second priority (ELP) andthe second priority to the first priority (ENP).

Threads in a parallel application often must wait for other threads tocomplete priority to resuming execution (i.e., a barrier operation). Thewait operation is completed through communication between the threads.This communication can be supported by an event that wakes a pausedthread, or by the waiting thread polling on a memory location. When athread is polling, it is wasting processing resources that could be usedby the thread that must finish its work to allow all threads to resumeproductive execution. The second or low priority queue 760 allows thewaiting threads to enter a low priority mode that will reduce theoverhead of the polling threads. This serves to reduce the threadexecution overhead of polling threads such that threads that mustcomplete productive work consume the majority of the availableprocessing resources.

A configuration register is used to determine the number of highpriority threads that are to be run for each low priority thread,illustrated in FIG. 30 as the low priority skip count, provided to thethread selection control circuitry 805, which selects a thread from thesecond priority queue 760 at predetermined intervals. As illustrated,thread selection control circuitry 805 decrements the skip count(register 842, multiplexer 844, and adder 846) until it is equal to zero(logic block 848), at which point the selection input of the threadselection multiplexer 785 toggles to select a thread from the second orlow priority queue 760.

FIG. 32 is a detailed block diagram of a representative embodiment ofdata path control circuitry 795 of an HTP 300. As mentioned above, oneor more of the HTPs 300 may also include data path control circuitry795, which is utilized to control access sizes (e.g., memory 125 loadrequests) over the first interconnection network 150 to manage potentialcongestion, providing adaptive bandwidth.

Application performance is often limited by the bandwidth available to aprocessor from memory. The performance limitation can be mitigated byensuring that only data that is needed by an application is brought intothe HTP 300. The data path control circuitry 795 automatically (i.e.,without user intervention) reduces the size of requests to main memory125 to reduce the utilization of the processor interface and memory 125subsystem.

As mentioned above, the compute resources of the system 100 may havemany applications using sparse data sets, with frequent accesses tosmall pieces of data distributed throughout the data set. As a result,if a considerable amount of data is accessed, much of it may be unused,wasting bandwidth. For example, a cache line may be 64 bytes, but notall of it will be utilized. At other times, it will be beneficial to useall available bandwidth, such as for efficient power usage. The datapath control circuitry 795 provides for dynamically adaptive bandwidthover the first interconnection network 150, adjusting the size of thedata path load to optimize performance of any given application, such asadjusting the data path load down to 8-32 bytes (as examples) based uponthe utilization of the receiving (e.g., response) channel of the firstinterconnection network 150 back to the HTP 300.

The data path control circuitry 795 monitors the utilization level onthe first interconnection network 150 and reduces the size of memory 125load (i.e., read) requests from the network interface circuitry 735 asthe utilization increases. In a representative embodiment, the data pathcontrol circuitry 795 performs a time-averaged weighting (time-averagedutilization block 764) of the utilization level of the response channelof the first interconnection network 150. If after a fixed period oftime (adjustment interval timer 762) the utilization is above athreshold (and the load request size is greater than the minimum), usingthreshold logic circuit 766 (having a plurality of comparators 882 andselection multiplexers 884, 886), then the size of load requests isreduced by the load request access size logic circuit 768 (generally bya power of 2 (e.g., 8 bytes) from the threshold logic circuit 766, usingminus increment 892), such that: either (a) fewer data packets 162 willbe included in the train of data packets 162, allowing that bandwidth tobe utilized for routing of data packets to another location or foranother process; or (b) memory 125 utilization is more efficient (e.g.,64 bytes are not requested when only 16 bytes will be utilized). Ifafter the fixed period of time the utilization is below a threshold (andthe load request size is less than the maximum), using threshold logiccircuit 766, then the size of the load request is increased by the loadrequest access size logic circuit 368, generally also by a power of 2(e.g., 8 bytes), using plus increment 888. The minimum and maximumvalues for the size of a load request can be user configured, however,the minimum size generally is the size of the issuing load instruction(e.g., the maximum operand size of the HTP 300, such as 8 bytes) and themaximum size is the cache line size (e.g., 32 or 64 bytes). In analternative embodiment, the data path control circuitry 795 can belocated at the memory controller 120, adapting to the bandwidth pressurefrom multiple HTPs 300.

FIG. 33 is a detailed block diagram of a representative embodiment ofsystem call circuitry 815 of an HTP 300 and host interface circuitry115. Representative system 100 embodiments allows a user mode onlycompute element, such as an HTP 300, to perform system calls,breakpoints and other privileged operations without running an operatingsystem, such as to open a file, print, etc. To do so, any of thesesystem operations are originated by an HTP 300 executing a user modeinstruction. The processor's instruction execution identifies that theprocessor must forward the request to a host processor 110 forexecution. The system request from the HTP 300 has the form of systemcall work descriptor packet sent to a host processor 110, and inresponse, the HTP 300 can receive system call return work descriptorpackets.

The system call work descriptor packet, assembled and transmitted by thepacket encoder 780, includes a system call identifier (e.g., a threadID, the core 705 number, a virtual address indicated by the programcounter, the system call arguments or parameters (which are typicallystored in the general-purpose registers 728), and return information.The packet is sent to a host interface 115 (SRAM FIFOs 864) that writesto and queues the system call work descriptor packets in a main memoryqueue, such as the illustrated DRAM FIFO 866 in host processor 110 mainmemory, increments a write pointer, and the host interface 115 furtherthen sends an interrupt to the host processor 110 for the host processor110 to poll for a system call work descriptor packet in memory. The hostprocessor's operating system accesses the queue (DRAM FIFO 866) entries,performs the requested operation and places return work descriptor datain a main memory queue (DRAM FIFO 868), and also may signal the hostinterface 115. The host interface 115 monitors the state of the returnqueue (DRAM FIFO 868) and when an entry exists, moves the data into anoutput queue (SRAM output queue 872) and formats a return workdescriptor packet with the work descriptor data provided and sends thereturn work descriptor packet to the HTP 300 which originated the systemcall packet.

The packet decoder 775 of the HTP 300 receives the return workdescriptor packet and places the returned arguments in thegeneral-purpose registers 728 as if the local processor (HTP 300)performed the operation itself. This transparent execution as viewed bythe application running on the user mode HTP 300 results in the abilityto use the same programming environment and runtime libraries that areused when a processor has a local operating system, and is highly usefulfor a wide variety of situations, such as program debugging, using aninserted break point.

The host interface 115, however, typically has limited FIFO space, whichcould be problematic when multiple HTPs 300 are utilized, each having alarge number of cores (e.g., 96), each of which may be running a largenumber of threads (e.g., 32/core). To avoid adding significant memory tothe host interface 115, the overall number of system calls which can besubmitted is limited, using a system call credit mechanism for each HTP300 and each processor core 705 within an HTP 300.

Each processor core 705 includes a first register 852, as part of thesystem call circuitry 815, which maintains a first credit count. Thesystem call circuitry 815, provided per HTP 300, includes a secondregister 858, which includes a second credit count, as a pool ofavailable credits. When a system call work descriptor packet isgenerated, if there are sufficient credits available in the firstregister 852, the system call work descriptor packet may be transmitted,and if not, the system call work descriptor packet is queued in thesystem call work descriptor packet table 862, potentially with othersystem call work descriptor packet from other processor cores 705 of thegiven HTP 300 (via multiplexer 854). If there are sufficient creditsavailable in the second register 858, providing an extra pool of creditsfor bursting of system calls and shared among all of the processor cores705 of the HTP 300, the next system call work descriptor packet may betransmitted, and otherwise is held in the table.

As those system call work descriptor packets are processed by the hostinterface 115 and read out of the FIFO 864, the host interface 115generates an acknowledgement back to the system call circuitry 815,which increments the credit counts per core in registers 856(illustrated as registers 856 ₀ and 856 ₁), which can in turn incrementthe first credit count in the first register 852, for each processorcore 705.

Alternatively, registers 856 may be utilized equivalently to a firstregister 852, without requiring the separate first register 852 percore, and instead maintaining the first count in the registers 856,again per core 705. As another alternative, all of the system call workdescriptor packets may be queued in the system call work descriptorpacket table 862, on a per core 705 basis, and transmitted when thatcore has sufficient first credit counts in its corresponding register856 or sufficient credits available in the second register 858.

A mechanism is also provided for thread state monitoring, to collect thestate of the set of threads running on an HTP 300 in hardware, whichallows a programmer to have the visibility into the workings of anapplication. With this feature, a host processor 110 can periodicallyaccess and store the information for later use in generating userprofiling reports, for example. With the provided visibility, aprogrammer can make changes to the application to improve itsperformance.

All thread state changes can be monitored and statistics kept on theamount of time in each state. The processor (110 or 300) that iscollecting the statistics provides a means for a separate, secondprocessor (110 or 300) to access and store the data. The data iscollected as the application is running such that a report can beprovided to an application analyst that shows the amount of time in eachstate reported on a periodic basis, which provides detailed visibilityon a running application for later use by an application analyst.

In accordance with the representative embodiments, which may beimplemented in hardware or software, all of the information pertainingto a thread is stored in the various registers of the thread memory 720,and can be copied and saved in another location on a regular basis. Acounter can be utilized to capture the amount of time any given threadspends in a selected state, e.g., a paused state. For example, the hostprocessor 110 can log or capture the current state of all threads andthread counters (amount of time spent in a state), or the differences(delta) between states and counts over time, and write it to a file orotherwise save it in a memory. Also for example, a program or thread maybe a barrier, in which all threads have to complete before anything elsecan start, and it is helpful to monitor which threads are in what stateas they proceed through various barriers or as they change state. Theillustrated code (below) is an example of simulator code which wouldexecute as hardware or be translatable to hardware:

InStateCount[N] - 6b InStateTimeStamp[N] - 64b InStateTotalTime[N] - 64benum ESimR5State { eR5Idle=0, eR5Low=1, eR5Normal=2, eR5PausedMem=3,eR5PausedEar=4,  eR5PausedEel=5, eR5PausedEer=6, eR5PausedEtr=7,eR5PausedEfj=8,  eR5PausedEfjAll=9, eR5PausedSys=10, eR5PausedEes=11 };// set state and collect statistics void setR5CtxState(SimR5HwCtx *pR5Ctx, SimR5HwCtx::ESimR5State state) { m_coreStats.m_coreInStateTotalTime[pR5Ctx−>m_r5State] +=   (getSimTime() − m_coreStats.m_coreInStateTime[pR5Ctx−>m_r5State]) *  m_coreStats.m_coreInStateCount[pR5Ctx−>m_r5State]; m_coreStats.m_coreInStateTime[pR5Ctx−>m_r5State] = getSimTime ( ); m_coreStats.m_coreInStateTotalTime[state] +=   (getSimTime ( ) −m_coreStats.m_coreInstateTime[state]) *  m_coreStats.m_coreInStateCount[state]; m_coreStats.m_coreInStateTime[state] = getSimTime ( ); m_coreStats.m_coreInStateCount[pR5Ctx−>m_r5State] −= 1; m_coreStats.m_coreInStateCount[state] += 1;  pR5Ctx−>m_r5State = state;} void incrementalStateStats(double incStateStats[HTP_R5_STATE_CNT]) { for (int state = 0; state < HTP_R5_STATE_CNT; state += 1) {  m_coreStats.m_coreInStateTotalTime[state] +=    (getSimTime ( ) −m_coreStats.m_coreInStateTime[state]) *   m_coreStats.m_coreInStateCount[state];  m_coreStats.m_coreInStateTime[state] = getSimTime ( );  incStateStats[state] += m_coreStats.m_coreInStateTotalTime[state] −   m_coreStats.m_coreInStatePrevTime[state];  m_coreStats.m_coreInStatePrevTime[state] =m_coreStats.m_coreInStateTotalTime[state];  } }

V. System Memory and Virtual Addressing:

The system 100 architecture provides a partitioned global address spaceacross all nodes within the system 100. Each node has a portion of theshared physical system 100 memory. The physical memory of each node ispartitioned into local private memory and global shared distributedmemory.

The local, private memory 125 of a node is accessible by all computeelements within that node. The compute elements within a nodeparticipate in a hardware-based cache coherency protocol. The hostprocessor 110 and HTPs 300 each maintain small data caches to acceleratereferences to private memory. The HTF 200 does not have a private memorycache (other than memory 325 and configuration memory 160), but ratherrelies on the memory subsystem cache to hold frequently accessed values.HTF 200 read and write requests are consistent at time of access. Thedirectory based cache coherency mechanism ensures that an HTF 200 readaccess obtains the most recently written value of memory and ensuresthat an HTF 200 write flushes dirty cache and invalidates sharedprocessor caches prior to writing the HTF 200 value to memory.

The distributed, shared memory of system 100 is accessible by allcompute elements within all nodes of the system 100, such as HTFs 200and HTPs 300. The system 100 processing elements do not have caches forshared memory, but rather may have read/write buffers with softwarecontrolled invalidation/flushing to minimize accesses to the same memoryline. The RISC-V ISA provides fence instructions that can be used toindicate a memory buffer invalidate/flush is needed. Similarly, the HTF200 supports write pause operations to indicate that all writeoperations to memory have completed. These write pause operations can beused to flush the read/write buffers.

An external host processor 110 will have its own system memory. Anapplication's node private virtual address space can include both hostprocessor system memory and system 100 node private memory. An externalhost processor 110 access to system memory can be kept consistentthrough the host processor's cache coherency protocol. External hostprocessor 110 access to system 100 node private memory across a PCIe orother communication interface 130 can be kept consistent by not allowingthe host processor 110 to cache the data. Other host to system 100 nodeinterfaces (i.e. CCIX or OpenCAPI) may allow the host processor to cachethe accessed data. Access to host processor system memory by system 100node compute elements across a PCIe interface can be kept consistent bynot allowing the compute elements to cache the data. Other host tosystem 100 node interfaces (i.e. CCIX or OpenCAPI) may allow the system100 compute elements to cache the data.

An external host processor 110 can access a node's private memorythrough the PCIe or other communication interface 130. These accessesare non-cacheable by the external host processor 110. Similarly, allnode processing elements may access an external processor's memorythrough the PCIe or other communication interface 130. It is normallymuch higher performance for the node's processing elements to access theexternal host's memory rather than have the host push data to the node.The node compute elements are architected to handle a higher number ofoutstanding requests and tolerate longer access latencies.

As mentioned above, in a representative embodiment, a system 100 processvirtual address space maps to physical memory on one or more system 100physical nodes. The system 100 architecture includes the concept of“virtual” nodes. System 100 virtual addresses include a virtual nodeidentifier. The virtual node identifier allows the requesting computeelement to determine if the virtual address refers to local node memoryor remote node memory. Virtual addresses that refer to local node memoryare translated to a local node physical address by the requestingcompute element. Virtual addresses that refer to remote node memory aresent to the remote node where, on entry to the node, the virtual addressis translated to a remote node physical address.

The concept of a virtual node allows a process to use the same set ofvirtual node identifiers independent of which physical nodes theapplication is actually executing on. The range of virtual nodeidentifiers for a process starts at zero and increases to the value N−1,where N is the number of virtual nodes in the process. The number ofvirtual nodes a process has is determined at runtime. The applicationmakes system call(s) to acquire physical nodes. The operating systemthen decides how many virtual nodes a process will have. The number ofphysical nodes given to a process is constrained by the number ofphysical nodes in the system 100. The number of virtual nodes may beequal to or larger than the number of physical nodes, but must be apower of two. Having a larger number of virtual nodes allows memory 125to be distributed across the physical nodes more uniformly. As anexample, if there are 5 physical nodes, and a process is setup to use 32virtual nodes, then shared, distributed memory can be distributed acrossthe physical nodes in increments of 1/32. The five nodes would have (7/32, 7/32, 6/32, 6/32, 6/32) of the total shared, distributed memoryper node. The uniformity of memory distribution also results in moreuniform bandwidth demand from the five nodes.

Having more virtual nodes than physical nodes within a process impliesthat multiple virtual nodes are assigned to a physical node. A node'scompute elements will each have a small table of local node virtual nodeIDs for a process. A maximum number of virtual node IDs per physicalnode IDs will exist. For example, a maximum number of virtual node IDsper physical node IDs may be eight, which allows the memory andbandwidth to be fairly uniform across the physical nodes without eachcompute element's virtual node ID table being too large.

The system 100 architecture has defined a single common virtual addressspace that is used by all compute elements. This common virtual addressspace is used by all threads executing on the system 100 computeelements (host processor 110, HTP 300 and HTF 200) on behalf of anapplication. The virtual-to-physical address translation process for ascalable multi-node system is carefully defined to ensure minimalperformance degradation as the system 100 scales. The system 100architecture has pushed the virtual-to-physical address translation tothe node where the physical memory resides as a solution to this scalingproblem. Performing the virtual-to-physical translation implies that thereferenced virtual address is transferred in the request packet that issent to the destination node. The request packet must be routed frominformation in the virtual address (since the physical address is notavailable until the packet reaches the destination node). Virtualaddresses are defined with the destination node ID embedded in theaddress. The exception is for external host virtual addresses to nodelocal, private memory. This exception is required due to x86 processorvirtual address space limitations.

The virtual address in current generations of x86 processors is 64 bitswide. However, for these 64-bit addresses, only the lower 48 bits areimplemented. The upper 16 bits must contain the sign extended value ofthe lower 48 bits. The virtual address space for an application runningon a standard Linux operating system is split between virtual addresswith the upper bits either all zero or all ones to be consistent withexisting processor restrictions. FIG. 37 shows the virtual address spaceformats supported by the system 100 architecture.

A system 100 virtual address is defined to support the full 64-bitvirtual address space. The upper three bits of the virtual address areused to specify the address format. The formats are defined in Table 31.

TABLE 31 Virtual Address Format ID Description 0 Format 0 is used to mapan external host’s main memory or local node’s private memory into thevirtual address space lower range of a Linux process. The upper 16 bitsare all zero to conform to the x86 virtual address restrictions. Theexternal host or a processing element (HTP or HTF) of the local node mayreference a virtual address using this format and access the externalhost’s main memory or local node’s private memory. Note access to thisaddress range always maps to the local node. Also note, the same virtualaddress, accessed by different nodes, will map to different physicalpages. Memory consistency in this address range is handled by hardware.Memory in this range is cacheable with exceptions. 1 Format 1 is used tomap private memory of a node into the virtual address space lower rangeof a Linux process. This format can be used by the internal RISC-Vprocessor. This private memory format has the local virtual node IDembedded in the virtual address. Embedding the Node ID in the virtualaddress allows nodes to detect that a non-local private memory virtualaddress is being accessed and raise an exception. Memory consistency inthis address range is handled by hardware. Memory in this range iscacheable with exceptions. 2 Format 2 is used to access shared,non-interleaved memory. Any HTP or HTF compute element executing athread for a Linux process may reference a virtual address associatedwith the partitioned global address space. Note that a host processor(internal or external) is not able to reference an address in thisformat range. Shared memory in this range is allocated as a singlecontiguous block of memory from a single node. Memory consistency inthis address range is managed by software and data accessed is notcacheable. 3 Format 3 is used to access shared, interleaved memory. AnyHTP or HTF compute element executing a thread of a Linux process mayreference a virtual address associated with the partitioned globaladdress space. Note that a host processor (external or internal) is notable to reference an address in this format range. Shared memory in thisrange is allocated across the largest power of two nodes that areparticipating in the process. The virtual address space is interleavedon a 4 Kbyte granularity across the nodes. Memory consistency in thisaddress range is managed by software and data accessed is not cacheable.4-5 Formats 4 and 5 are not used. 6 Format 6 is used to map privatememory of a node into the virtual address space upper range of a Linuxprocess. This format can be used by the internal RISC-V processor. Thisprivate memory format has the local node ID embedded in the virtualaddress. Embedding the Node ID in the virtual address allows nodes todetect that a non-local private memory virtual address is being accessedand raise an exception. Memory consistency in this address range ishandled by hardware. Memory in this range is cacheable with exceptions.7 Format 7 is used to map an external host’s main memory or local node’sprivate memory into the virtual address space upper range of a Linuxprocess. The upper 16 bits are all ones to conform to the x86 virtualaddress restrictions. The external host or a processing element (HTP orHTF) of the local node may reference a virtual address using this formatand access the external host’s main memory or local node’s privatememory. Note access to this address range always maps to the local node.Also note, the same virtual address, accessed by different nodes, willmap to different physical pages. Memory consistency in this addressrange is handled by a hardware. Memory in this range is cacheable withexceptions.

The exceptions referred to in Table 31 can occur due to two situations:(1) a private address is sent to a remote node HTP or HTF computeelement as an argument to the sent call or return operation, or (2) adata structure in shared memory is created with a pointer to privatememory.

FIG. 38 shows the translation process for each virtual address format.Referring to FIGS. 37 and 38 :

(a) Format 0 and 7 are used by the external host processor 110 and bythe local node host processor 110, HTP 300 and HTF 200 compute elementsto access external host memory as well as local node private memory. Thesource compute element of the memory request translates the virtualaddress to a physical address.

(b) Format 1 and 6 are used by the local node host processor 110, HTP300 and HTF 200 compute elements to access local node private memory, aswell as external host memory. It should be noted that use of this formatallows a remote node device to validate that the local node privatememory reference is indeed intended for the local node. The situationwhere this becomes valuable is if a local node's private virtual addressis used by a remote node. The remote node can compare the embedded nodeID with the local node ID and detect the memory reference error. Itshould be noted that this detection capability is not available withformat 0.

(c) Format 2 is used by all node host processor 110, HTP 300 and HTF 200compute elements to access non-interleaved, distributed shared memory.Allocations to this memory format will allocate a contiguous block ofphysical memory on the node where the allocation occurs. Each node of aprocess is numbered with a virtual node ID starting at zero andincreasing to as many nodes as in the process. The virtual-to-physicaladdress translation first translates the virtual node ID in the virtualaddress to a physical node ID. The node ID translation occurs at thesource node. Once translated, the physical node ID is used to route therequest to the destination node. It should be noted that both the GlobalSpace ID (GSID) and virtual address are sent in the packet to thedestination node. Once at the destination node, the remote nodeinterface receives the request packet and translates the virtual addressto the local node's physical address.

(d) Format 3 is used by all node host processor 110, HTP 300 and HTF 200compute elements to access interleaved, distributed shared memory.Allocations to this memory format will allocate a block of memory oneach node participating in the interleave (the largest power of twonodes in the process). References to this format are interleaved on a 4Kbyte granularity (the actual interleave granularity is beinginvestigated). The first step of the translation process is to swap thevirtual node ID in the virtual address from the lower bits to the upperbits (swapped into the position starting with bit 48). After the node IDis swapped, then the virtual node ID is translated to the physical nodeID. The node ID swapping and translation occurs on the source node. Thephysical node ID is used to route the request to the destination node.It should be noted that both the Global Space ID (GSID) and virtualaddress are sent in the packet to the destination node. Once at thedestination node, the remote node interface receives the request packet,and translates the virtual address to the local node physical address.

(e) Formats 4 and 5 are not utilized, and in a representativeembodiment, these formats are illegal and will cause a referenceexception.

Numerous advantages of the representative embodiments are readilyapparent. The representative apparatus, system and methods provide for acomputing architecture capable of providing high performance and energyefficient solutions for compute-intensive kernels, such as forcomputation of Fast Fourier Transforms (FFTs) and finite impulseresponse (FIR) filters used in sensing, communication, and analyticapplications, such as synthetic aperture radar, 5G base stations, andgraph analytic applications such as graph clustering using spectraltechniques, machine learning, 5G networking algorithms, and largestencil codes, for example and without limitation.

Significantly, the various representative embodiments provide amulti-threaded, coarse-grained configurable computing architecturecapable of being configured for any of these various applications, butmost importantly, also capable of self-scheduling, dynamicself-configuration and self-reconfiguration, conditional branching,backpressure control for asynchronous signaling, ordered threadexecution and loop thread execution (including with data dependencies),automatically starting thread execution upon completion of datadependencies and/or ordering, providing loop access to privatevariables, providing rapid execution of loop threads using a reenterqueue, and using various thread identifiers for advanced loop execution,including nested loops.

Also, the representative apparatus, system and method provide for aprocessor architecture capable of self-scheduling, significant parallelprocessing and further interacting with and controlling a configurablecomputing architecture for performance of any of these variousapplications.

As used herein, a “processor core” 705 may be any type of processorcore, and may be embodied as one or more processor cores configured,designed, programmed or otherwise adapted to perform the functionalitydiscussed herein. As used herein, a “processor” 110 may be any type ofprocessor, and may be embodied as one or more processors configured,designed, programmed or otherwise adapted to perform the functionalitydiscussed herein. As the term processor is used herein, a processor 110or 300 may include use of a single integrated circuit (“IC”), or mayinclude use of a plurality of integrated circuits or other componentsconnected, arranged or grouped together, such as controllers,microprocessors, digital signal processors (“DSPs”), array processors,graphics or image processors, parallel processors, multiple coreprocessors, custom ICs, application specific integrated circuits(“ASICs”), field programmable gate arrays (“FPGAs”), adaptive computingICs, associated memory (such as RAM, DRAM and ROM), and other ICs andcomponents, whether analog or digital. As a consequence, as used herein,the term processor or controller should be understood to equivalentlymean and include a single IC, or arrangement of custom ICs, ASICs,processors, microprocessors, controllers, FPGAs, adaptive computing ICs,or some other grouping of integrated circuits which perform thefunctions discussed herein, with associated memory, such asmicroprocessor memory or additional RAM, DRAM, SDRAM, SRAM, MRAM, ROM,FLASH, EPROM or E²PROM. A processor 110 or 300, with associated memory,may be adapted or configured (via programming, FPGA interconnection, orhard-wiring) to perform the methodology of the invention, as discussedherein. For example, the methodology may be programmed and stored, in aprocessor 300 with its associated memory (and/or memory 125) and otherequivalent components, as a set of program instructions or other code(or equivalent configuration or other program) for subsequent executionwhen the processor 110 or 300 is operative (i.e., powered on andfunctioning). Equivalently, when the processor 300 may implemented inwhole or part as FPGAs, custom ICs and/or ASICs, the FPGAs, custom ICsor ASICs also may be designed, configured and/or hard-wired to implementthe methodology of the invention. For example, the processor 110 or 300may be implemented as an arrangement of analog and/or digital circuits,controllers, microprocessors, DSPs and/or ASICs, collectively referredto as a “processor” or “controller”, which are respectively hard-wired,programmed, designed, adapted or configured to implement the methodologyof the invention, including possibly in conjunction with a memory 125.

The memory 125, 325, which may include a data repository (or database),may be embodied in any number of forms, including within any computer orother machine-readable data storage medium, memory device or otherstorage or communication device for storage or communication ofinformation, currently known or which becomes available in the future,including, but not limited to, a memory integrated circuit (“IC”), ormemory portion of an integrated circuit (such as the resident memorywithin a processor 130 or processor IC), whether volatile ornon-volatile, whether removable or non-removable, including withoutlimitation RAM, FLASH, DRAM, SDRAM, SRAM, MRAM, FeRAM, ROM, EPROM orE²PROM, or any other form of memory device, such as a magnetic harddrive, an optical drive, a magnetic disk or tape drive, a hard diskdrive, other machine-readable storage or memory media such as a floppydisk, a CDROM, a CD-RW, digital versatile disk (DVD) or other opticalmemory, or any other type of memory, storage medium, or data storageapparatus or circuit, which is known or which becomes known, dependingupon the selected embodiment. The memory 125, 325 may be adapted tostore various look up tables, parameters, coefficients, otherinformation and data, programs or instructions (of the software of thepresent invention), and other types of tables such as database tables.

As indicated above, the processor 110, 300 is hard-wired or programmed,using software and data structures of the invention, for example, toperform the methodology of the present invention. As a consequence, thesystem and related methods of the present invention, including thevarious instructions, may be embodied as software which provides suchprogramming or other instructions, such as a set of instructions and/ormetadata embodied within a non-transitory computer readable medium,discussed above. In addition, metadata may also be utilized to definethe various data structures of a look up table or a database. Suchsoftware may be in the form of source or object code, by way of exampleand without limitation. Source code further may be compiled into someform of instructions or object code (including assembly languageinstructions or configuration information). The software, source code ormetadata of the present invention may be embodied as any type of code,such as C, C++, Matlab, SystemC, LISA, XML, Java, Brew, SQL and itsvariations (e.g., SQL 99 or proprietary versions of SQL), DB2, Oracle,or any other type of programming language which performs thefunctionality discussed herein, including various hardware definition orhardware modeling languages (e.g., Verilog, VHDL, RTL) and resultingdatabase files (e.g., GDSII). As a consequence, a “construct”, “programconstruct”, “software construct” or “software”, as used equivalentlyherein, means and refers to any programming language, of any kind, withany syntax or signatures, which provides or can be interpreted toprovide the associated functionality or methodology specified (wheninstantiated or loaded into a processor or computer and executed,including the processor 110, 300, for example).

The software, metadata, or other source code of the present inventionand any resulting bit file (object code, database, or look up table) maybe embodied within any tangible, non-transitory storage medium, such asany of the computer or other machine-readable data storage media, ascomputer-readable instructions, data structures, program modules orother data, such as discussed above with respect to the memory 125, 325,e.g., a floppy disk, a CDROM, a CD-RW, a DVD, a magnetic hard drive, anoptical drive, or any other type of data storage apparatus or medium, asmentioned above.

The communication interface(s) 130 are utilized for appropriateconnection to a relevant channel, network or bus; for example, thecommunication interface(s) 130 may provide impedance matching, driversand other functions for a wireline or wireless interface, may providedemodulation and analog to digital conversion for a wireless interface,and may provide a physical interface, respectively, for the processor110, 300 and/or memory 125, with other devices. In general, thecommunication interface(s) 130 are used to receive and transmit data,depending upon the selected embodiment, such as program instructions,parameters, configuration information, control messages, data and otherpertinent information.

The communication interface(s) 130 may be implemented as known or maybecome known in the art, to provide data communication between the HTF200 and/or processor 110, 300 and any type of network or externaldevice, such as wireless, optical, or wireline, and using any applicablestandard (e.g., one of the various PCI, USB, RJ 45, Ethernet (FastEthernet, Gigabit Ethernet, 300ase-TX, 300ase-FX, etc.), IEEE 802.11,Bluetooth, WCDMA, WiFi, GSM, GPRS, EDGE, 3G and the other standards andsystems mentioned above, for example and without limitation), and mayinclude impedance matching capability, voltage translation for a lowvoltage processor to interface with a higher voltage control bus,wireline or wireless transceivers, and various switching mechanisms(e.g., transistors) to turn various lines or connectors on or off inresponse to signaling from processor 130. In addition, the communicationinterface(s) 130 may also be configured and/or adapted to receive and/ortransmit signals externally to the system 100, such as throughhard-wiring or RF or infrared signaling, for example, to receiveinformation in real-time for output on a display, for example. Thecommunication interface(s) 130 may provide connection to any type of busor network structure or medium, using any selected architecture. By wayof example and without limitation, such architectures include IndustryStandard Architecture (ISA) bus, Enhanced ISA (EISA) bus, Micro ChannelArchitecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, SANbus, or any other communication or signaling medium, such as Ethernet,ISDN, T1, satellite, wireless, and so on.

The present disclosure is to be considered as an exemplification of theprinciples of the invention and is not intended to limit the inventionto the specific embodiments illustrated. In this respect, it is to beunderstood that the invention is not limited in its application to thedetails of construction and to the arrangements of components set forthabove and below, illustrated in the drawings, or as described in theexamples. Systems, methods and apparatuses consistent with the presentinvention are capable of other embodiments and of being practiced andcarried out in various ways.

Although the invention has been described with respect to specificembodiments thereof, these embodiments are merely illustrative and notrestrictive of the invention. In the description herein, numerousspecific details are provided, such as examples of electroniccomponents, electronic and structural connections, materials, andstructural variations, to provide a thorough understanding ofembodiments of the present invention. One skilled in the relevant artwill recognize, however, that an embodiment of the invention can bepracticed without one or more of the specific details, or with otherapparatus, systems, assemblies, components, materials, parts, etc. Inother instances, well-known structures, materials, or operations are notspecifically shown or described in detail to avoid obscuring aspects ofembodiments of the present invention. In addition, the various Figuresare not drawn to scale and should not be regarded as limiting.

Reference throughout this specification to “one embodiment”, “anembodiment”, or a specific “embodiment” means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention and notnecessarily in all embodiments, and further, are not necessarilyreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics of any specific embodiment of the presentinvention may be combined in any suitable manner and in any suitablecombination with one or more other embodiments, including the use ofselected features without corresponding use of other features. Inaddition, many modifications may be made to adapt a particularapplication, situation or material to the essential scope and spirit ofthe present invention. It is to be understood that other variations andmodifications of the embodiments of the present invention described andillustrated herein are possible in light of the teachings herein and areto be considered part of the spirit and scope of the present invention.

For the recitation of numeric ranges herein, each intervening numberthere between with the same degree of precision is explicitlycontemplated. For example, for the range of 6-9, the numbers 7 and 8 arecontemplated in addition to 6 and 9, and for the range 6.0-7.0, thenumber 6.0, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9, and 7.0 areexplicitly contemplated. In addition, every intervening sub-range withinrange is contemplated, in any combination, and is within the scope ofthe disclosure. For example, for the range of 5-10, the sub-ranges 5-6,5-7, 5-8, 5-9, 6-7, 6-8, 6-9, 6-10, 7-8, 7-9, 7-10, 8-9, 8-10, and 9-10are contemplated and within the scope of the disclosed range.

It will also be appreciated that one or more of the elements depicted inthe Figures can also be implemented in a more separate or integratedmanner, or even removed or rendered inoperable in certain cases, as maybe useful in accordance with a particular application. Integrally formedcombinations of components are also within the scope of the invention,particularly for embodiments in which a separation or combination ofdiscrete components is unclear or indiscernible. In addition, use of theterm “coupled” herein, including in its various forms such as “coupling”or “couplable”, means and includes any direct or indirect electrical,structural or magnetic coupling, connection or attachment, or adaptationor capability for such a direct or indirect electrical, structural ormagnetic coupling, connection or attachment, including integrally formedcomponents and components which are coupled via or through anothercomponent.

With respect to signals, we refer herein to parameters that “represent”a given metric or are “representative” of a given metric, where a metricis a measure of a state of at least part of the regulator or its inputsor outputs. A parameter is considered to represent a metric if it isrelated to the metric directly enough that regulating the parameter willsatisfactorily regulate the metric. A parameter may be considered to bean acceptable representation of a metric if it represents a multiple orfraction of the metric.

Furthermore, any signal arrows in the drawings/Figures should beconsidered only exemplary, and not limiting, unless otherwisespecifically noted. Combinations of components of steps will also beconsidered within the scope of the present invention, particularly wherethe ability to separate or combine is unclear or foreseeable. Thedisjunctive term “or”, as used herein and throughout the claims thatfollow, is generally intended to mean “and/or”, having both conjunctiveand disjunctive meanings (and is not confined to an “exclusive or”meaning), unless otherwise indicated. As used in the description hereinand throughout the claims that follow, “a”, “an”, and “the” includeplural references unless the context clearly dictates otherwise. Also asused in the description herein and throughout the claims that follow,the meaning of “in” includes “in” and “on” unless the context clearlydictates otherwise.

The foregoing description of illustrated embodiments of the presentinvention, including what is described in the summary or in theabstract, is not intended to be exhaustive or to limit the invention tothe precise forms disclosed herein. From the foregoing, it will beobserved that numerous variations, modifications and substitutions areintended and may be effected without departing from the spirit and scopeof the novel concept of the invention. It is to be understood that nolimitation with respect to the specific methods and apparatusillustrated herein is intended or should be inferred. It is, of course,intended to cover by the appended claims all such modifications as fallwithin the scope of the claims.

It is claimed:
 1. A system comprising: a first, interconnection network;a second, packet network; a third, synchronous network; a processorcoupled to the first, interconnection network, the processor configuredto generate and transmit a first work descriptor data packet on thefirst, interconnection network; a plurality of configurable circuitsarranged in an array, each configurable circuit of the plurality ofconfigurable circuits coupled to the second, packet network and to thethird, synchronous network, each configurable circuit of the pluralityof configurable circuits comprising: a configurable computation circuit;and a configuration memory coupled to the configurable computationcircuit, the configuration memory comprising: a first instruction memoryconfigured to store a first plurality of configuration instructions toconfigure the configurable computation circuit; and a second instructionmemory configured to store a second plurality of configurationinstructions or instruction indices for selection of a configurationinstruction of the first plurality of configuration instructions fromthe first instruction memory and for selection of a synchronous networkinput from the third, synchronous network for receipt of theconfiguration instruction or a configuration instruction index fromanother configurable circuit of the plurality of configurable circuitsof the array.
 2. The system of claim 1, further comprising: a dispatchinterface circuit coupled to the first, interconnection network andcoupled through the second, packet network to the plurality ofconfigurable circuits, the dispatch interface circuit configured, inresponse to receiving the first descriptor data packet through thefirst, interconnection network, to generate and transmit on the second,packet network one or more data and control packets to one or moreconfigurable circuits of the array of the plurality of configurablecircuits to configure the one or more configurable circuits forexecution of a selected computation.
 3. The system of claim 1, whereinthe processor comprises: a core control circuit comprising: a threadcontrol memory comprising a thread identifier register configured tostore a plurality of thread identifiers, a program count registerconfigured to store an initial program count, and a data cache or ageneral-purpose register configured to store a received argument; anexecution queue coupled to the thread control memory; and a controllogic and thread selection circuit coupled to the execution queue and tothe thread control memory, the control logic and thread selectioncircuit configured, in response to a second work descriptor data packethaving the initial program count and the received argument, to assign athread identifier of the first plurality of thread identifiers to anexecution thread and to place the thread identifier in the executionqueue, the control logic and thread selection circuit further configuredto select the thread identifier for execution of the execution thread;and a processor core coupled to the core control circuit, the processorcore configured to execute at least one instruction using the receivedargument for the execution thread, the at least one instructioncorresponding to the initial program count.
 4. The system of claim 3,wherein the core control circuit further comprises: an interconnectionnetwork interface coupled to the first, interconnection network and tothe thread control memory, the interconnection network interfaceconfigured to receive the second work descriptor data packet and todecode the second work descriptor data packet into the initial programcount for the execution thread and the received argument for theexecution thread.
 5. The system of claim 3, wherein the core controlcircuit further comprises: a network command queue coupled to theprocessor core; and an interconnection network interface coupled to thefirst, interconnection network and to the network command queue; whereinthe processor core is configured to execute a fiber create instructionto generate and provide one or more commands to the network commandqueue, and wherein in response to the one or more commands in thenetwork command queue, the interconnection network interface is furtherconfigured to generate and transmit a plurality of call work descriptorpackets to the array of the plurality of configurable circuits forexecution of a corresponding plurality of execution threads.
 6. Thesystem of claim 3, wherein thread control memory further comprises athread state register, the thread state register configured to store avalid status or a paused status for each thread identifier of the firstplurality of thread identifiers, and wherein the control logic andthread selection circuit is further configured to place the threadidentifier in the execution queue when the thread identifier has thevalid status, the control logic and thread selection circuit furtherconfigured, for as long as the valid status of the thread identifierremains, to periodically select the thread identifier in the executionqueue for execution of an instruction of the execution thread by theprocessor core, and the control logic and thread selection circuitfurther configured to pause execution of the execution thread by notreturning the thread identifier to the execution queue when the threadidentifier has the paused status.
 7. The system of claim 1, wherein eachconfigurable circuit, of the array of the plurality of configurablecircuits, further comprises: a configuration memory multiplexer coupledto the first instruction memory and to the second instruction memory,the configuration memory multiplexer having a selection input, whereinwhen the selection input has a first setting, the current data pathconfiguration instruction is selected from the first plurality of datapath configuration instructions of the first instruction memory using aselected data path configuration instruction or instruction index fromthe second instruction memory, and wherein when the selection input hasa second setting, the second setting different from the first setting,the current data path configuration instruction is selected using a datapath configuration instruction or instruction index received from thesynchronous network input.
 8. The system of claim 1, wherein the secondplurality of configuration instructions or instruction indices furthercomprises instructions or instruction indices for selection of a nextconfiguration instruction or instruction index for output to a nextconfigurable circuit of the plurality of configurable circuits.
 9. Thesystem of claim 8, wherein each configurable circuit, of the pluralityof configurable circuits, further comprises: a conditional logic circuitcoupled to the configurable computation circuit, the conditional logiccircuit configured, depending upon an output from the configurablecomputation circuit, to modify the next configuration instruction orinstruction index to output a modified next configuration instruction orinstruction index to the next configurable circuit.
 10. The system ofclaim 1, wherein each configurable circuit, of the plurality ofconfigurable circuits, further comprises: a packet network input queuecoupled to the second, packet network, the packet network input queueconfigured to receive a first message or packet from the second, packetnetwork; a packet network output queue coupled to the second, packetnetwork, the packet network output queue configured to transfer a secondmessage or packet for routing on the second, packet network; and whereinthe system further comprises: a stop signal communication line coupledto each of the configurable circuits of the plurality of configurablecircuits; and a flow control circuit coupled to the packet networkoutput queue of each configurable circuit of the plurality ofconfigurable circuits of the array, the flow control circuit configuredto generate a stop signal on the stop signal communication line when apredetermined threshold has been reached in the packet network outputqueue; wherein, in response to receipt of the stop signal, eachconfigurable circuit of the plurality of configurable circuits of thearray is configured to pause execution while continuing to receive oneor more third messages or packets routed from the second, packetnetwork.
 11. The system of claim 1, wherein each configurable circuit,of the plurality of configurable circuits, further comprises: a controlcircuit coupled to the configurable computation circuit, the controlcircuit comprising: a plurality of control registers configured to storea completion table having a plurality of thread identifiers, having acorresponding data completion count for each thread identifier of theplurality of thread identifiers and, for each thread identifier for acurrent thread of the plurality of thread identifiers, having anidentification of a last iteration or another thread identifieridentifying a next thread for execution following execution of thecurrent thread; and a thread control circuit configured to self-scheduleand queue a selected thread for execution when, for the threadidentifier for the selected thread, the corresponding data completioncount indicates completion of one or more data dependencies and thethread identifier for the selected thread is identified as the nextthread for execution.
 12. The system of claim 1, wherein theinterconnection network is configured to use a split header and payloadconfiguration for pipelining a plurality of communications or fordelayed payload switching, and wherein the interconnection network isfurther configured to use a single header to route a plurality of datapayloads as a sequential data burst.
 13. The system of claim 1, whereineach configurable circuit, of the plurality of configurable circuits,further comprises: a plurality of synchronous network inputs; and aplurality of synchronous network outputs; wherein the third, synchronousnetwork is configured to couple each synchronous network input of theplurality of synchronous network inputs to a corresponding synchronousnetwork output, of the plurality of synchronous network outputs, of oneor more adjacent configurable circuits of the plurality of configurablecircuits of the array to form a plurality of direct connections betweenadjacent configurable circuits of the array, and wherein the third,synchronous network further comprises at least one pass throughconnection between adjacent configurable circuits of the array.
 14. Thesystem of claim 13, wherein one or more of the plurality of theconfigurable circuits are coupled in series through the first,synchronous network to form a first synchronous domain and one or moreof the plurality of the configurable circuits are coupled in seriesthrough the first, synchronous network to form a second synchronousdomain, the plurality of configurable circuits configured to perform aplurality of computations using the first and second synchronousdomains, and the plurality of configurable circuits further configuredto generate and transmit a plurality of control messages over thesecond, packet network, the plurality of control messages comprising oneor more completion messages and one or more continue messages.
 15. Asystem comprising: a first, interconnection network; a second, packetnetwork; a third, synchronous network; a processor coupled to the first,interconnection network, the processor configured to execute a fibercreate instruction and to generate and transmit a plurality of workdescriptor data packets on the first, interconnection network; a networkinterface circuit coupled to the first, interconnection network, thenetwork interface circuit configured to receive one or more workdescriptor data packets of the plurality of work descriptor datapackets; a plurality of configurable circuits arranged in an array, eachconfigurable circuit of the plurality of configurable circuits coupledto the second, packet network and to the third, synchronous network,each configurable circuit of the plurality of configurable circuitscomprising: a configurable computation circuit; and a configurationmemory coupled to the configurable computation circuit, theconfiguration memory comprising: a first instruction memory configuredto store a first plurality of configuration instructions to configurethe configurable computation circuit; and a second instruction memoryconfigured to store a second plurality of configuration instructions orinstruction indices for selection of a configuration instruction of thefirst plurality of configuration instructions from the first instructionmemory and for selection of a network input from the third, synchronousnetwork for receipt of the configuration instruction or a configurationinstruction index from another configurable circuit of the plurality ofconfigurable circuits of the array; and a dispatch interface circuitcoupled to the network interface circuit and coupled through the second,packet network to the plurality of configurable circuits of the array,the dispatch interface circuit configured, in response to receiving theone or more work descriptor data packets from the network interfacecircuit, to generate and transmit on the second, packet network one ormore data and control packets to one or more configurable circuits ofthe array of the plurality of configurable circuits to configure the oneor more configurable circuits for execution of a corresponding pluralityof execution threads.
 16. The system of claim 15, wherein the processorcomprises: a core control circuit comprising: a thread control memorycomprising a thread identifier register configured to store a pluralityof thread identifiers, a program count register configured to store aninitial program count, and a data cache or a general-purpose registerconfigured to store a received argument; an execution queue coupled tothe thread control memory; and a control logic and thread selectioncircuit coupled to the execution queue and to the thread control memory,the control logic and thread selection circuit configured, in responseto a second work descriptor data packet having the initial program countand the received argument, to assign a thread identifier of the firstplurality of thread identifiers to an execution thread and to place thethread identifier in the execution queue, the control logic and threadselection circuit further configured to select the thread identifier forexecution of the execution thread; and a processor core coupled to thecore control circuit, the processor core configured to execute at leastone instruction using the received argument for the execution thread,the at least one instruction corresponding to the initial program count.17. The system of claim 15, wherein each configurable circuit, of theplurality of configurable circuits, further comprises: a control circuitcoupled to the configurable computation circuit, the control circuitcomprising: a plurality of control registers configured to store acompletion table having a plurality of thread identifiers, having acorresponding data completion count for each thread identifier of theplurality of thread identifiers and, for each thread identifier for acurrent thread of the plurality of thread identifiers, having anidentification of a last iteration or another thread identifieridentifying a next thread for execution following execution of thecurrent thread; and a thread control circuit configured to self-scheduleand queue a selected thread for execution when, for the threadidentifier for the selected thread, the corresponding data completioncount indicates completion of one or more data dependencies and thethread identifier for the selected thread is identified as the nextthread for execution.
 18. The system of claim 15, wherein theinterconnection network is configured to use a split header and payloadconfiguration for pipelining a plurality of communications or fordelayed payload switching, and wherein the interconnection network isfurther configured to use a single header to route a plurality of datapayloads as a sequential data burst; and wherein each configurablecircuit, of the plurality of configurable circuits, further comprises: aplurality of synchronous network inputs; and a plurality of synchronousnetwork outputs; wherein the third, synchronous network is configured tocouple each synchronous network input of the plurality of synchronousnetwork inputs to a corresponding synchronous network output, of theplurality of synchronous network outputs, of one or more adjacentconfigurable circuits of the plurality of configurable circuits of thearray to form a plurality of direct connections between adjacentconfigurable circuits of the array, and wherein the third, synchronousnetwork further comprises at least one pass through connection betweenadjacent configurable circuits of the array.
 19. The system of claim 15,wherein the second plurality of configuration instructions orinstruction indices further comprises instructions or instructionindices for selection of a next configuration instruction or instructionindex for output to a next configurable circuit of the plurality ofconfigurable circuits; and wherein each configurable circuit, of theplurality of configurable circuits, further comprises: a conditionallogic circuit coupled to the configurable computation circuit, theconditional logic circuit configured, depending upon an output from theconfigurable computation circuit, to modify the next configurationinstruction or instruction index to output a modified next configurationinstruction or instruction index to the next configurable circuit of thearray.
 20. A system comprising: a first, interconnection network havinga first plurality of communication lines for transmitting a packetheader and a second plurality of communication lines for transmitting adata payload; a second, packet network; a third, synchronous network; aprocessor coupled to the first, interconnection network, the processorcomprising: a processor core configured to execute a plurality ofinstructions; and a core control circuit coupled to the processor core,the core control circuit configured to generate and transmit a pluralityof work descriptor data packets on the first, interconnection network; aplurality of configurable circuits arranged in an array, eachconfigurable circuit of the plurality of configurable circuits coupledto the second, packet network and to the third, synchronous network,each configurable circuit of the plurality of configurable circuitscomprising: a plurality of synchronous network outputs; a plurality ofsynchronous network inputs, each synchronous network input of theplurality of synchronous network inputs coupled to a correspondingsynchronous network output, of the plurality of synchronous networkoutputs, of one or more adjacent configurable circuits of the pluralityof configurable circuits of the array to form a plurality of directconnections between adjacent configurable circuits of the array; aconfigurable computation circuit; and a configuration memory coupled tothe configurable computation circuit, the configuration memorycomprising: a first instruction memory configured to store a firstplurality of configuration instructions to configure the configurablecomputation circuit; and a second instruction memory configured to storea second plurality of configuration instructions or instruction indicesfor selection of a configuration instruction of the first plurality ofconfiguration instructions from the first instruction memory and forselection of a synchronous network input from the plurality ofsynchronous network inputs for receipt of the configuration instructionor a configuration instruction index from another configurable circuitof the plurality of configurable circuits of the array; and a dispatchinterface circuit coupled to the first, interconnection network andcoupled through the second, packet network to the plurality ofconfigurable circuits of the array, the dispatch interface circuitconfigured, in response to receiving the one or more work descriptordata packets, to generate and transmit on the second, packet network oneor more data and control packets to one or more configurable circuits ofthe array of the plurality of configurable circuits to configure the oneor more configurable circuits for execution of a corresponding pluralityof execution threads.